Pith. sign in

REVIEW 3 major objections 4 minor 57 references

Token-level contrastive disagreement between reference-guided and vanilla probabilities is a correctness signal that shapes RL advantages better than entropy, resolves the zero-advantage problem, and subsumes on-policy distillation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:34 UTC pith:RHOUFHQS

load-bearing objection A practical token-level credit-assignment signal for RLVR with solid empirical gains, but the paper's load-bearing correctness proxy is only validated for negative disagreement. the 3 major comments →

arxiv 2607.14614 v1 pith:RHOUFHQS submitted 2026-07-16 cs.LG cs.AIcs.CL

Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

Weiwen Xu , Jia Liu , Hou Pong Chan , Long Li , Deng Cai , Min Chen , Hao Zhang This is my paper
classification cs.LG cs.AIcs.CL
keywords contrastive policy optimizationtoken-level credit assignmentadvantage shapingRL with verifiable rewardsentropy vs correctnesson-policy distillationzero-advantage problemLLM reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning with verifiable rewards typically tells a model only whether its whole answer was right or wrong, and uses output entropy to add finer guidance. This paper argues that entropy cannot tell productive uncertainty from confusion, and replaces it with a contrastive disagreement: for each token, the log-ratio of its probability when the model is shown the reference answer versus when it generates normally. The core claim is that this ratio tracks whether that token leads to a correct completion, so it can reshape the per-response advantage (the score that says how much better or worse a sampled solution was than average) at the token level. If right, this gives finer credit assignment, puts zero-advantage groups back into the learning signal, and reveals on-policy distillation as a special case of the same correctness-driven objective. The paper reports consistent gains over entropy-based shaping on math reasoning with preserved out-of-domain generalization.

Core claim

CPO's central discovery is that the token-level contrastive disagreement δ_t(x,y) = log[π(y_t|x,y*,y_<t)/π(y_t|x,y_<t)] is a correctness-aware signal. By Bayes' rule, the log-ratio of the ideal correctness-conditioned distribution to the prior equals log g(x,y_<t,y_t) − log Z(x,y_<t), a strictly increasing function of the position-wise correctness probability g. Conditioning the same model on the reference answer y* approximates this ideal posterior (Assumption C.7), so negative disagreement reliably flags tokens whose correctness probability is below the prior average. CPO shapes the group-relative advantage by adding a clipped multiple of δ_t, preserving the trajectory-level advantage's di

What carries the argument

The central object is the contrastive disagreement ratio (a named object): δ_t = log[π(y_t | x, y*, y_<t) / π(y_t | x, y_<t)], the log-ratio of reference-guided to vanilla next-token probabilities. Its theoretical status comes from a Bayes-rule identity showing the ideal version of this ratio equals log g − log Z, monotone in the token's oracle correctness probability. The practical machinery is the shaping rule of Eq. (8), which adds α·δ to the trajectory advantage and clips to prevent reversal, plus the reference-guided prompting that instantiates the posterior without an external model.

Load-bearing premise

The load-bearing premise is that conditioning the model on the reference answer produces a distribution that matches what the model would assign if it knew whether the completion is correct—an approximation (Assumption C.7) that is only empirically justified and holds strongly for wrong tokens but weakly for right ones.

What would settle it

Take a set of model rollouts with step-level correctness labels (from a process reward model or human annotation), compute δ_t at each token, and check whether tokens with negative δ_t are systematically more likely to be incorrect than baseline tokens across multiple domains (math, code, science). If on any domain the negative-disagreement tokens are not more likely to be incorrect, the correctness-alignment claim fails. A complementary test: condition on a deliberately wrong or random 'reference' answer; if the advantage gains persist, the signal is not actually tracking correctness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Token-level credit assignment becomes finer-grained: correct and incorrect tokens within a correct or incorrect solution receive differentiated signal, improving hard multi-step reasoning.
  • Zero-advantage groups (all rollouts equally correct or equally incorrect) are no longer wasted; the contrastive disagreement provides a usable learning signal there.
  • On-policy distillation is recovered as a special case when the posterior is an external teacher, placing RL and distillation under one correctness-driven objective.
  • Training retains a higher entropy plateau and longer responses, sustaining exploration and improving Pass@K rather than only top-1 sampling.
  • Because the signal is self-contrastive (same model, no external preference), out-of-domain performance is preserved or improved relative to entropy-based baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own empirical check shows the signal is strongly reliable for negative disagreement but flat for positive disagreement; this suggests CPO's practical power may come mostly from mistake detection, and a sharper positive signal might come from better reference prompting or a temperature-calibrated posterior.
  • The unification claim implies a testable design space: interpolating between external-teacher and reference-conditioned posteriors should trade off generalization (teacher) and immediate correctness (reference); the paper shows one data point of combining them, so a sweep over mixing weights is a natural next experiment.
  • If the reference-conditioned posterior can be replaced by a critic or process-level correctness signal, CPO could extend beyond verifiable rewards to open-ended reasoning without ground-truth answers—an extension the paper mentions but does not test.
  • The 20% training overhead for the extra forward pass could be amortized via asynchronous or cached posterior estimates; whether the signal tolerates staleness is an open empirical question.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. CPO replaces entropy-based advantage shaping in GRPO with a token-level contrastive disagreement δ_t between a reference-guided posterior and the vanilla policy. The paper argues via a Bayes-rule identity (Appendix C) that the ideal disagreement is monotone in token-level correctness, then uses reference-conditioned prompting as a practical proxy and shapes group advantages with clipped token-level terms. It further claims that on-policy distillation is a special case of CPO and that CPO resolves the zero-advantage problem. Experiments on Qwen2.5-Math-7B and Qwen3-Base-4B on in-domain math and out-of-domain benchmarks report large gains over entropy-based baselines, with ablations and a controlled zero-advantage experiment.

Significance. The paper has a clean theoretical setup, a practically motivated proposal, and a commendable amount of experimental work: full prompts, hyperparameters, ablation studies, and a controlled zero-advantage setting are all provided. If the contrastive-disagreement signal truly indicates token-level correctness, CPO would be a meaningful advance over entropy-based RLVR and would provide an elegant unifying perspective on on-policy distillation. However, the load-bearing proxy assumption (C.7) is only empirically validated for negative disagreement; the positive-disagreement side, which drives CPO's correct-response branch, is flat in the validation study and therefore unverified. The significance of the paper is real but conditional on closing this gap or appropriately restricting the claims.

major comments (3)
  1. [Assumption C.7 / §2.1.2] Theorem C.8 is a Bayes-rule identity about the ideal disagreement, not about the observed δ_t. The step from ideal to observed disagreement is entirely through Assumption C.7, which is justified only by the empirical proxy check in §2.1.2. That check is asymmetric: negative disagreement strongly lowers Accuracy@16, but positive disagreement is flat near the exp(δ)=1 baseline. Under C.8/C.9, positive δ should predict above-baseline correctness; flatness is evidence against the proxy for exactly the tokens that CPO's α_+ branch upweights in correct responses. The 'capability ceiling' explanation conflates a null correlation with saturation. Please add a more direct validation of the positive side (e.g., threshold sweep, banded analysis, a stronger backbone, or another estimator of g), or restrict the correctness-awareness claim to negative/incorrect-leaning tokens. As written, the central
  2. [Tables 1–2, F.1] The headline comparison to entropy-based baselines lacks error bars. Only CPO, GRPO, and DAPO have 3-run variance (Table 8); the entropy-intervened baselines are single runs. Some differences are modest, and Table 8 shows large run-to-run swings (e.g., DAPO AIME2024 ±10.3). Without variance for Entropy-Tokens, Entropy-Adv, EM-RL-token, RL-ZVP, and IB-reg, the claim of 'substantially outperforms entropy-based RLVR methods' is not fully supported. Please report repeated runs or confidence intervals for the main baselines, or at least justify why single runs are sufficient.
  3. [F.2 / Abstract] The claim that CPO 'resolves the zero-advantage problem' is stronger than the controlled evidence. The isolated zero-advantage experiment in Table 9 shows only a +3.2% average improvement over RL-ZVP, which is modest and based on one backbone, one data split, and one evaluation protocol. The larger gains in the full setting conflate CPO's overall advantage shaping with the zero-advantage mechanism. Please either temper the 'resolves' language or provide additional evidence that the signal on zero-advantage groups is qualitatively different from entropy-based shaping (e.g., gradient statistics, learning curves, or additional controlled datasets).
minor comments (4)
  1. [§2.1.2, Eq. (6)] The first-threshold procedure and the choice of δ are not fully described; the empirical validation could be sensitive to this selection. Please report the results for different δ values and include variance over the 16 resampled continuations, since Figure 2 is the main support for Assumption C.7.
  2. [§3.3, Figure 4] OPD and the CPO variants appear to be single runs; small differences (e.g., 71.2 vs 71.5 for the in-domain average) may not be meaningful. Please add error bars or explicitly state that these comparisons are not statistically significant.
  3. [Abstract / §1] The statement that on-policy distillation is 'a special case' of CPO is made informally. Since this is a claimed contribution, please provide a formal proposition or precise objective mapping (e.g., which δ-based objective reduces to which OPD loss) in Appendix C.
  4. [Appendix E, Figure 6] α+ and α− are both set to 0.025, but Figure 6 varies only the α+ : α− ratio. Absolute scale matters in Eq. (8) because of the clipping. Please include a sensitivity analysis or justify why only the ratio matters.

Circularity Check

2 steps flagged

The central 'disagreement indicates correctness' theorem is a definitional Bayes identity; the observable signal inherits correctness-awareness only through the empirically one-sided proxy Assumption C.7.

specific steps
  1. self definitional [Sec. 2.1.1 / Eq. (3)-(4); Appendix C, Lemma C.5, Theorem C.8]
    "By Bayes’ rule, this distribution is regarded as a reweighting of the prior by correctness probability: π̃post(yt|x,y<t) = π(yt|x,y<t)g(x,y<t,yt)/Z(x,y<t) ... Taking logarithms yields: log π̃post(yt|x,y<t)/π(yt|x,y<t) = log g(x,y<t,yt) − log Z(x,y<t)."

    π̃post is defined as P(Yt=yt | x, y<t, C), i.e., by conditioning on the correctness event C. Therefore its log-ratio against the prior π is, by construction, the log of the correctness probability g up to a prefix constant. Theorem C.8 states a consequence of this definition and cannot fail; it has no independent empirical content. The claim becomes a statement about the observable δ only when Assumption C.7 is added, so the 'theoretical result' reduces to the definition of the ideal posterior plus a proxy assumption.

  2. self definitional [Appendix C, Assumption C.7 and Corollary C.9; Sec. 2.1.1 Eq. (5)]
    "Since y⋆ encodes the ground-truth that determines correctness and the prompt also asks the model to refine, conditioning on such prompt serves as a practical proxy for conditioning on the correctness event C, i.e., πpost(·|x,y⋆,y<t)≈π̃post(·|x,y<t)."

    With this assumption in place, δt(x,y)=log πpost/π is by construction a monotone increasing function of g(x,y<t,yt) (via Eq. 4 and Eq. 5). The paper's central assertion that 'contrastive disagreement reliably indicates token-level correctness' is thus built into the assumption, not derived. The only supporting evidence is the proxy check in Sec. 2.1.2, which is one-sided: positive disagreement (the branch used to shape correct responses) stays flat near baseline, so the empirical justification itself does not independently confirm the assumed monotonicity for the tokens CPO rewards.

full rationale

The paper's formal derivation (Lemma C.5, Theorem C.8) is a mathematically correct Bayes-rule identity: once the ideal posterior is defined as conditioning on the correctness event, the ideal log-ratio equals log g minus a prefix constant. This is true by definition and carries no empirical content about the actual observed disagreement. The entire scientific claim that the observable δt indicates token-level correctness rests on Assumption C.7, which asserts that the reference-conditioned posterior approximates the correctness-conditioned ideal posterior. That assumption is an empirical bridge, not a derivation; its justification in Sec. 2.1.2 is noisy (16 continuations, one model, one dataset) and asymmetric, with positive disagreement showing flat accuracy near baseline, which is exactly the regime used by CPO's alpha_plus branch for correct responses. The RL experiments themselves are independent: CPO is evaluated on held-out in-domain and out-of-domain benchmarks against standard baselines, with hyperparameters (alpha_plus, alpha_minus) fixed rather than fitted to the test sets, and no fitted constants are renamed as predictions. Self-citations to the authors' prior works ([5], [16], [40]) appear only in related-work or illustrative contexts and are not load-bearing for the main claim. Overall, the circularity is partial: the theoretical justification reduces to a definition plus an empirically under-supported proxy assumption, but the method's empirical comparison retains independent content, so a moderate score of 4 is appropriate rather than a higher score reserved for results forced entirely by construction or by self-citation chains.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claim rests on one hand-chosen hyperparameter (α) and the unverified proxy Assumption C.7. No new physical or model entities are introduced; the 'correctness-informed posterior' is a conceptual object, not a postulated entity.

free parameters (2)
  • Combination strengths α_+, α_- = α_+ = α_- = 0.025
    Hand-selected hyperparameters in Eq. (8) controlling how strongly token-level disagreement reshapes advantages. The ablation varies only their ratio, not the joint scale, so reported gains are conditional on this choice.
  • Disagreement threshold δ in validation = exp(δ) from 0.1 to 10 in Fig. 2
    Used only in the Sec. 2.1.2 empirical validation to define fork positions t*. It is not a training parameter, but the evidence that δ indicates correctness is threshold-dependent and shows asymmetry between positive and negative δ.
axioms (4)
  • ad hoc to paper π_post(·|x,y∗,y_<t) ≈ π̃_post(·|x,y_<t) (Assumption C.7)
    This is the load-bearing bridge between the Bayes identity and the claim that observed δ tracks token correctness. It is justified only by an empirical proxy check, not derived. If it fails, CPO's signal is not correctness-aligned.
  • domain assumption C is an oracle-defined, model-agnostic binary correctness event (Definition C.1)
    Needed to define g and the ideal correctness-conditioned posterior. It assumes a reference answer uniquely determines correctness for any completion, which is reasonable for MATH-style verifiable rewards but not for all RLVR settings.
  • domain assumption The reference-guided prompt induces a distribution 'more correctness-informed' than the vanilla policy
    Underlies the OPD-unification discussion and the choice of π_post. It is not formally verified and depends on prompt design and model instruction-following.
  • standard math Standard probability calculus (Bayes' rule, marginalization) in Lemma C.5
    Used to derive Eq. (4) and Theorem C.8; not in question.

pith-pipeline@v1.3.0-alltime-deepseek · 18995 in / 12857 out tokens · 138498 ms · 2026-08-02T01:34:10.505536+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization." pith.science (2026). https://pith.science/paper/RHOUFHQS

@misc{pith2026260714614,
  author       = {Pith},
  title        = {Pith review of: Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHOUFHQS}},
  note         = {Machine review of arXiv:2607.14614}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.

Figures

Figures reproduced from arXiv: 2607.14614 by Deng Cai, Hao Zhang, Hou Pong Chan, Jia Liu, Long Li, Min Chen, Weiwen Xu.

Figure 1
Figure 1. Figure 1: Top: A student spots his homework mistake by comparing his work to the reference answer. Bottom: An LLM’s response generation probabilities shift when guided by a reference answer. valuable exploration from detrimental errors. This competing effects have led to contradictory design choices across existing methods: some reward high-entropy tokens in correct responses [13], while others instead favor low-ent… view at source ↗
Figure 3
Figure 3. Figure 3: The illustration of CPO. Prior and reference-guided posterior likelihoods are contrasted to shape token-level advantages. provides a good estimate of the ideal log-ratio in Eq. (4). Importantly, this holds regardless of whether the rollout y is correct or not—the correctness probability g(x, y<t, yt) is defined over future continuations, not individual trajectories. Full derivations are provided in Appendi… view at source ↗
Figure 2
Figure 2. Figure 2: Impact of contrastive disagreement on token correctness (Accuracy@16). We empirically validate Eq. (5), showing that contrastive disagreement δt(x, y) reliably in￾dicates token-level correctness g(x, y<t, yt). Since g(x, y<t, yt) represents the expected cor￾rectness averaging over all possible future con￾tinuations from token yt, it is intractable to com￾pute exactly. We approximate it with Accu￾racy@16, i… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of different posterior de￾signs (πpost) under a unified CPO framework. We study different instantiations of πpost under the CPO framework in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Word clouds of 100 tokens with top absolute disagreement (CPO) and entropy: (a) CPO - Entropy: tokens only detected by CPO; (b) CPO ∩ Entropy: tokens detected by both; and (c) Entropy - CPO: tokens detected only by entropy. Contrastive disagreement vs. entropy. While CPO is not derived from entropy, prior analyses show that it strongly impacts entropy. To un￾derstand their relationship and key differences,… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 19 linked inside Pith

  1. [1]

    R., Geist, M., and Bachem, O

    Agarwal, R., Vieillard, N., Zhou, Y ., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On- policy distillation of language models: Learning from self-generated mistakes. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=3zKtaqxLhW

    2024

  2. [2]

    The unreasonable effectiveness of entropy minimization in llm reasoning.arXiv preprint arXiv:2505.15134, 2025

    Agarwal, S., Zhang, Z., Yuan, L., Han, J., and Peng, H. The unreasonable effectiveness of entropy minimization in llm reasoning.arXiv preprint arXiv:2505.15134, 2025. URL https://arxiv.org/abs/2505.15134

    Pith/arXiv arXiv 2025

  3. [3]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.Advances in neu- ral information processing systems, 33:1877–1901, 2020. URLhttps://papers.nips.cc/ paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    1901

  4. [4]

    X., Zhang, Z., and Wei, F

    Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, W. X., Zhang, Z., and Wei, F. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758, 2025. URL https://arxiv.org/abs/2506.14758

    Pith/arXiv arXiv 2025

  5. [5]

    K., Chen, G., Xu, W., Luu, A

    Chia, Y . K., Chen, G., Xu, W., Luu, A. T., Poria, S., and Bing, L. Reasoning paths optimiza- tion: Learning to reason and explore from diverse paths. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 16763–16780, Miami, Florida, USA, November

    2024

  6. [6]

    X., and Wen, J.-R

    Deng, J., Chen, J., Chen, Z., Zhao, W. X., and Wen, J.-R. Decomposing the entropy-performance exchange: The missing keys to unlocking effective reinforcement learning.arXiv preprint arXiv:2508.02260, 2025. URLhttps://arxiv.org/abs/2508.02260

    Pith/arXiv arXiv 2025

  7. [7]

    Deepseek-r1: incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081): 633–638, September 2025

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081): 633–638, September 2025. URLhttp://dx.doi.org/10.1038/s41586-025-09422-z

    doi:10.1038/s41586-025-09422-z 2025

  8. [9]

    Measuring mathematical problem solving with the MATH dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URLhttps://openreview.net/forum?id=7Bywt2mQsCe

    2021

  9. [10]

    K., Guestrin, C., et al

    Hübotter, J., Lübeck, F., Behric, L., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Buening, T. K., Guestrin, C., et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    Pith/arXiv arXiv 2026

  10. [11]

    F., and Joty, S

    Jiao, F., Qin, C., Liu, Z., Chen, N. F., and Joty, S. Learning planning-based reasoning by trajectories collection and process reward synthesizing. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 334–350, Miami, Florida, USA, 10 November 2024. Association for Computational Linguistics. URL https://aclanthology...

    2024

  11. [12]

    Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., Liu, A., Dziri, N., Lyu, X., Gu, Y ., Malik, S., Graf, V ., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y ., Dasigi, P., and Hajishirzi, H. Tulu 3: Pushing frontiers in open language model post-training. InS...

    2025

  12. [13]

    V ., Jeon, M., Vu, K., Lai, V ., and Yang, E

    Le, T.-L. V ., Jeon, M., Vu, K., Lai, V ., and Yang, E. No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping. arXiv preprint arXiv:2509.21880, 2025. URLhttps://arxiv.org/abs/2509.21880

    arXiv 2025

  13. [14]

    Revisiting llm reasoning via information bottleneck

    Lei, S., Cheng, Z., Jia, K., and Tao, D. Revisiting llm reasoning via information bottleneck. arXiv preprint arXiv:2507.18391, 2025. URLhttps://arxiv.org/abs/2507.18391

    Pith/arXiv arXiv 2025

  14. [15]

    RED: Unleashing token-level rewards from holistic feedback via reward redistribution

    Li, J., Li, L., Chang, T.-W., Kuang, K., Chen, L., Zhou, J., and Yang, C. RED: Unleashing token-level rewards from holistic feedback via reward redistribution. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 4993–5022, Suzhou, China, November 2025. Association for Computational Linguistics. URL https: //aclant...

    2025

  15. [16]

    Can we further elicit reasoning in LLMs? critic-guided planning with retrieval-augmentation for solving challenging tasks

    Li, X., Xu, W., Zhao, R., Jiao, F., Joty, S., and Bing, L. Can we further elicit reasoning in LLMs? critic-guided planning with retrieval-augmentation for solving challenging tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 25589–25604, Vienna, Austria, July 2025. Association fo...

    2025

  16. [17]

    Let’s verify step by step

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=v8L0pN6EOi

    2024

  17. [18]

    Ravr: Reference-answer-guided variational reasoning for large language models.arXiv preprint arXiv:2510.25206, 2025

    Lin, T., Zhao, X., Zhang, X., Long, R., Xu, Y ., Jiang, Z., Su, W., and Zheng, B. Ravr: Reference-answer-guided variational reasoning for large language models.arXiv preprint arXiv:2510.25206, 2025. URLhttps://arxiv.org/abs/2510.25206

    arXiv 2025

  18. [19]

    and Lab, T

    Lu, K. and Lab, T. M. On-policy distillation.Thinking Machines Lab: Connection- ism, 2025. doi: 10.64434/tml.20251026. URL https://thinkingmachines.ai/blog/ on-policy-distillation

    doi:10.64434/tml.20251026 2025

  19. [20]

    American mathematics competitions (AMC), 2023

    Mathematical Association of America. American mathematics competitions (AMC), 2023. URLhttps://maa.org/. Accessed: 2025-12-23

    2023

  20. [21]

    American invitational mathematics examination (AIME),

    Mathematical Association of America. American invitational mathematics examination (AIME),

  21. [22]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=Ti67584b98

    2024

  22. [23]

    Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/ 1707.06347

    Pith/arXiv arXiv 2017

  23. [24]

    Accessed: 2025-12-23

    URLhttps://maa.org/. Accessed: 2025-12-23

    2025

  24. [25]

    On entropy control in llm-rl algorithms.arXiv preprint arXiv:2509.03493, 2025

    Shen, H. On entropy control in llm-rl algorithms.arXiv preprint arXiv:2509.03493, 2025. URL https://arxiv.org/abs/2509.03493

    arXiv 2025

  25. [26]

    Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026

    Shenfeld, I., Damani, M., Hübotter, J., and Agrawal, P. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026. 11

    Pith/arXiv arXiv 2026

  26. [27]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. URLhttps://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  27. [28]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Team, K., Bai, Y ., Bao, Y ., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y ., Chen, Y ., Chen, Y ., et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025. URL https://arxiv.org/abs/2507.20534

    Pith/arXiv arXiv 2025

  28. [29]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. URLhttps://arxiv.org/abs/2307.09288

    Pith/arXiv arXiv 2023

  29. [30]

    Espo: Entropy importance sampling policy optimization.arXiv preprint arXiv:2512.00499, 2025

    Sheng, Y ., Huang, Y ., Liu, S., Zhang, H., and Zeng, A. Espo: Entropy importance sampling policy optimization.arXiv preprint arXiv:2512.00499, 2025. URL https://arxiv.org/ abs/2512.00499

    arXiv 2025

  30. [31]

    Arbitrary entropy policy optimization: Entropy is controllable in reinforcement fine-tuning.arXiv preprint arXiv:2510.08141, 2025

    Wang, C., Li, Z., Bai, J., Zhang, Y ., Cui, S., Zhao, Z., and Wang, Y . Arbitrary entropy policy optimization: Entropy is controllable in reinforcement fine-tuning.arXiv preprint arXiv:2510.08141, 2025. URLhttps://arxiv.org/abs/2510.08141

    Pith/arXiv arXiv 2025

  31. [32]

    Harnessing uncertainty: Entropy-modulated policy gradients for long-horizon llm agents.arXiv preprint arXiv:2509.09265, 2025

    Wang, J., Liu, J., Fu, Y ., Li, Y ., Wang, X., Lin, Y ., Yue, Y ., Zhang, L., Wang, Y ., and Wang, K. Harnessing uncertainty: Entropy-modulated policy gradients for long-horizon llm agents.arXiv preprint arXiv:2509.09265, 2025. URLhttps://arxiv.org/abs/2509.09265

    Pith/arXiv arXiv 2025

  32. [33]

    and Karkhanis, D

    Walder, C. and Karkhanis, D. T. Pass@k policy optimization: Solving harder reinforcement learning problems. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=W6WC6047X2

    2025

  33. [34]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning

    Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X.-H., Yang, J., Zhang, Z., Liu, Y ., Yang, A., Zhao, A., Yue, Y ., Song, S., Yu, B., Huang, G., and Lin, J. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. InThe Thirty-ninth Annual Conference on Neural Information Processi...

    2025

  34. [35]

    MMLU-pro: A more robust and challenging multi-task language understanding benchmark

    Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,...

    2024

  35. [36]

    Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations

    Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y ., Chen, D., Wu, Y ., and Sui, Z. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439, Bangkok, Thailand, August 2024. Association for Computational...

    2024

  36. [37]

    Williams, R. J. and Peng, J. Function optimization using connectionist reinforcement learning algorithms.Connection Science, 3(3):241–268, 1991. URL https://doi.org/10.1080/ 09540099108946587

    1991

  37. [38]

    Quantile advantage estimation for entropy-safe reasoning.arXiv preprint arXiv:2509.22611, 2025

    Wu, J., Huang, K., Wu, J., Zhang, A., Wang, X., and He, X. Quantile advantage estimation for entropy-safe reasoning.arXiv preprint arXiv:2509.22611, 2025. URL https://arxiv.org/ abs/2509.22611

    arXiv 2025

  38. [39]

    Openclaw-rl: Train any agent simply by talking.arXiv preprint arXiv:2603.10165, 2026

    Wang, Y ., Chen, X., Jin, X., Wang, M., and Yang, L. Openclaw-rl: Train any agent simply by talking.arXiv preprint arXiv:2603.10165, 2026

    Pith/arXiv arXiv 2026

  39. [40]

    Reasons to reject? aligning language models with judgments

    Xu, W., Cai, D., Zhang, Z., Lam, W., and Shi, S. Reasons to reject? aligning language models with judgments. InFindings of the Association for Computational Linguistics: ACL 2024, pp. 12288–12304, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URLhttps://aclanthology.org/2024.findings-acl.730/. 12

    2024

  40. [41]

    Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024. URLhttps://arxiv.org/abs/2409.12122

    Pith/arXiv arXiv 2024

  41. [42]

    A., Osten- dorf, M., and Hajishirzi, H

    Wu, Z., Hu, Y ., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Osten- dorf, M., and Hajishirzi, H. Fine-grained human feedback gives better rewards for lan- guage model training.Advances in Neural Information Processing Systems, 36:59008– 59033, 2023. URL https://papers.neurips.cc/paper_files/paper/2023/file/ b8c90b65739ae8417e61eadb521f63d...

    2023

  42. [43]

    Self-distillation bridges distribution gap in language model fine-tuning

    Yang, Z., Pang, T., Feng, H., Wang, H., Chen, W., Zhu, M., and Liu, Q. Self-distillation bridges distribution gap in language model fine-tuning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1028– 1043, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL ht...

    2024

  43. [44]

    Tree of thoughts: Deliberate problem solving with large language models

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y ., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. In Oh, A., Nau- mann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.),Advances in Neural Information Processing Systems, volume 36, pp. 11809–11822. Curran Associates, Inc., 2023. URL ...

    2023

  44. [45]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. URL https: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  45. [46]

    Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    Yue, Y ., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y ., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=4OsgYD7em5

    2025

  46. [47]

    Knowlogic: A benchmark for commonsense reasoning via knowledge-driven data synthesis

    Zhan, W., Wang, Y ., Hu, N., Xiao, L., Ma, J., Qin, Y ., Li, Z., Yang, Y ., Deng, S., Ding, J., et al. Knowlogic: A benchmark for commonsense reasoning via knowledge-driven data synthesis. arXiv preprint arXiv:2503.06218, 2025. URLhttps://arxiv.org/abs/2503.06218

    Pith/arXiv arXiv 2025

  47. [48]

    DAPO: An open-source LLM reinforcement learning system at scale

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., YuYue, Dai, W., Fan, T., Liu, G., Liu, J., Liu, L., Liu, X., Lin, H., Lin, Z., Ma, B., Sheng, G., Tong, Y ., Zhang, C., Zhang, M., Zhang, R., Zhang, W., Zhu, H., Zhu, J., Chen, J., Chen, J., Wang, C., Yu, H., Song, Y ., Wei, X., Zhou, H., Liu, J., Ma, W.-Y ., Zhang, Y .-Q., Yan, L., Wu, Y ., and Wang, M. DAP...

    2025

  48. [49]

    R., Kailkhura, B., Lai, F., Zhao, J., and Chen, B

    Zheng, H., Zhou, Y ., Bartoldson, B. R., Kailkhura, B., Lai, F., Zhao, J., and Chen, B. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=x5lITYXmW2

    2025

  49. [50]

    First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017, 2025

    Zheng, T., Xing, T., Gu, Q., Liang, T., Qu, X., Zhou, X., Li, Y ., Wen, Z., Lin, C., Huang, W., et al. First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017, 2025. URL https://arxiv.org/abs/2507.07017. 13 A Limitations The primary computational overhead of CPO stems from the additional reference-guided forward pass required to compute th...

    Pith/arXiv arXiv 2025

  50. [51]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y ., Men, R., Yang, A., et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025. URL https://arxiv.org/abs/2507.18071

    Pith/arXiv arXiv 2025

  51. [54]

    Output only the first-person thought process, no preamble or summary

  52. [55]

    Simulate real-time problem-solving tone, avoid meta-commentary

  53. [56]

    Don’t mention or imply knowing the reference answer (avoid ’according to the answer...’ etc.)

  54. [57]

    Show complete reasoning path with intermediate steps, verification, and error-correction, not just restatement

  55. [58]

    Put final answer in \boxed{}<|im_end|> <|im_start|>assistant Table 6: Prompt used in multi-turn gold-shot generation. <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user [question]<|im_end|> <|im_start|>assistant [reference answer]<|im_end|> <|im_start|>user [question] Please reason step by step, and put your final answer within \bo...

    2004

  56. [2024]

    URL https://aclanthology.org/2024

    Association for Computational Linguistics. URL https://aclanthology.org/2024. findings-emnlp.977/

    2024

  57. [2025]

    URLhttps://arxiv.org/abs/2510.10150

    Pith/arXiv arXiv