Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

Selective expert guidance at high-uncertainty tokens lets reinforcement learning explore effectively without collapsing into imitation of expert trajectories.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:29 UTC pith:7HYUXWVS

load-bearing objection A real empirical contribution with a misleading 'selective guidance' story: MENTOR's entropy weighting is dense, not sparse, but the gains are consistent and the idea is worth engaging. the 3 major comments →

arxiv 2510.04140 v2 pith:7HYUXWVS submitted 2025-10-05 cs.AI cs.CL

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Zishang Jiang , Jinyi Han , Tingyun Li , Xinyi Wang , Sihang Jiang , Jiaqing Liang , Zhaoqian Dai , Shuguang Ma
show 2 more authors
Fei Yu Yanghua Xiao
This is my paper
classification cs.AI cs.CL
keywords reinforcement learningRLVRexpert guidanceentropyexplorationLLM reasoningmixed policytoken-level optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that in reinforcement learning from verifiable rewards, an expert model should not be imitated across entire reasoning chains; instead, expert guidance is only needed at 'critical decision points' — tokens where the policy is most uncertain, measured by high entropy. To test this, the authors build MENTOR, which samples from a mixture of the policy's own distribution and an expert's distribution, with the mixture weight rising with token entropy. Across three base models and six math and reasoning benchmarks, they report that this selective guidance beats both standard on-policy reinforcement learning and methods that imitate full expert trajectories, while keeping response diversity high. The larger point is that a weaker model can absorb an expert's reasoning strategy without surrendering its own exploration, which could make RLVR work on smaller models.

Core claim

MENTOR's central claim is that token-level entropy of the learner's own next-token distribution marks where expert help is needed. At each decoding step the paper forms a mixed distribution πmix(·|q,y<t) = (1−wt)πθ(·|q,y<t) + wtπ*(·|q,y<t) with wt = min(1, Ht/γp), where Ht is the entropy of πθ and γp is the p-quantile of entropies in the batch. High-entropy positions therefore lean on the expert; low-entropy positions stay with the learner. The paper reports that this selective guidance outperforms both plain on-policy GRPO and methods that imitate full expert trajectories across three base models and six benchmarks, and that it slows entropy collapse and raises pass@32 diversity. The interp

What carries the argument

The mechanism is the token-level mixed-policy distribution πmix(·|q,y<t) = (1−wt)πθ(·|q,y<t) + wtπ*(·|q,y<t), with wt = min(1, Ht/γp). This interpolation is the load-bearing object: it injects expert likelihood only where the learner is uncertain, turning a global imitation problem into a local one. The paper couples this with accelerated rollout via speculative sampling (draft from πθ, verify against πmix) and a modified group-relative advantage that rewards mixed-policy rollouts only when they beat the on-policy group mean, with the mixing weight α annealed to zero over training.

Load-bearing premise

The load-bearing premise is that a token's on-policy entropy reliably marks where an expert's help would change the reasoning fork; if high entropy only reflects stylistic uncertainty, mixing in the expert at those spots is just noise and the claimed balance would not arise.

What would settle it

Run MENTOR with the mixed-policy tokens selected by a random schedule instead of by entropy, holding the number of expert-guided positions fixed. If accuracy and diversity gains survive, entropy is not the cause; inspect a sample of high-entropy positions to see whether they are actual reasoning forks.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RLVR can be applied to smaller base models without losing effectiveness, because expert guidance fills gaps in the learner's exploration.
  • Models trained this way keep higher response entropy through training and end at a wider support set, which the paper links to stronger final performance.
  • The reasoning patterns learned transfer to out-of-domain tasks, indicating the trained model is not just overfitting to the training distribution.
  • Because the expert is only queried at selected tokens, the method remains computationally practical via the speculative-sampling acceleration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An ablation that replaces entropy-based token selection with random selection at the same rate is not reported; if the gains persist, entropy would not be the mechanism, and if they vanish, entropy-based selection is doing the work.
  • The method assumes the expert distribution is more competent specifically at high-entropy decision points. If the expert's style diverges from the learner's, mixing at those points could import expert biases instead of task-relevant reasoning, so the benefit may depend on the expert's compatibility.
  • The α-annealing schedule suggests a natural curriculum — expert-guided exploration early, self-driven refinement later. One testable extension is making the entropy threshold itself adaptive over training, rather than fixed at a batch percentile.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MENTOR, a training framework for RLVR in LLMs that mixes the current policy with an expert policy at the token level, with the mixture weight proportional to token entropy, and uses a modified GRPO advantage for mixed rollouts. The central claim is that providing expert guidance only at high-entropy 'critical decision points' preserves exploration diversity while improving effectiveness, unlike full-trajectory imitation. Experiments on Qwen2.5-3B/7B and LLaMA3.1-8B across math and out-of-domain benchmarks show consistent gains over on-policy GRPO, LUFFY, and QuestA, with additional analyses of entropy dynamics, reasoning-token occurrence, and pass@32.

Significance. If the mechanism worked as claimed, this would be a practical contribution: it offers a principled way to inject expert knowledge into RLVR without collapsing diversity, and the implementation includes an unbiased speculative-sampling acceleration with a self-contained proof. The experimental comparison is broad (three backbones, multiple benchmarks), the code is released, and the training-dynamics analyses are informative. However, the precise mechanism — selective gating at critical decision points — is not what the equations implement, and the formal analysis in Section 2.2/Appendix A.1 does not apply to the trained objective. These issues are load-bearing for the paper's central claim and need to be addressed before the contribution is established.

major comments (3)
  1. [Sec. 3.1, Eq. (6), Algorithm 1] The central claim that MENTOR provides expert guidance 'only at critical decision points' is not implemented by Eq. (6). With wt = min(1, Ht/γp) and γp the 95th percentile of token entropies in the batch (Appendix B), every token with Ht>0 receives a positive expert-mixture weight; essentially all tokens in natural text have Ht>0, so the expert contributes at every decoding step. The 'sparsity' argued in Section 3.1 is therefore not a property of the sampling distribution. Speculative sampling in Algorithm 1 only accelerates sampling from this dense πmix; it does not introduce hard gating, and Appendix A.2 correctly proves unbiasedness for the dense πmix. The diversity argument that 'expert guidance is restricted to only a few positions' is unsupported. A hard-gate ablation (e.g., wt=0 below the threshold) or an explicit reframing as dense soft weighting with entropy-scaled strength is r
  2. [Sec. 2.2 and Appendix A.1] The formal analysis claims to derive necessary conditions for high-quality exploration in RLVR, but it proves concentration of the maximum-entropy distribution under a reward constraint, using the Maximum Entropy Principle in Eq. (12). The actual system trains with GRPO and, as stated in Appendix B, uses KL coefficient 0 and entropy loss 0. Thus the theorem concerns a different objective and cannot establish the claim about support contraction for the algorithm used. Either replace this analysis with one for the actual objective or clearly label it as an analogy/illustration; as written, the 'formal analysis' contribution overstates its target.
  3. [Sec. 3.1, Sec. 4.5] The identification of high-entropy tokens with 'critical decision forks' is not directly evidenced. The paper cites prior work (Wang et al., 2025) but does not analyze MENTOR's own rollouts to show that expert intervention at those positions is what drives the gains. The entropy threshold is a batch-level quantile, not a semantic property of the token in context. Without an ablation comparing entropy-gated mixing against (a) uniform-position mixing with the same average weight, or (b) a different entropy threshold, the observed improvements could come from dense soft mixing or from the mixed-policy advantage in Eq. (11) rather than from the entropy-based selection.
minor comments (4)
  1. [Abstract, Sec. 1] Grammar and redundancy: 'enables models capture' should be 'enables models to capture'; 'provide guidance only at critical decision points' appears twice with 'only'.
  2. [Sec. 4.1 baselines list] The entry for QuestA is duplicated: '(4)QuestA(Li et al., 2025): (4)QuestA...'.
  3. [Algorithm 1] The line 'Compute πmix(·|q, y≤n) ← (1−wt)πθ(·|q, y≤n) + wtπ∗(·|q, y≤n)' omits the candidate prefix ỹ<t in the conditioning; it should read πmix(·|q, y≤n, ỹ<t) for clarity.
  4. [Appendix B] Typo: 'Export Model Details' should be 'Expert Model Details'. Also, Eq. (11) uses R_range before it is defined; define it earlier in the main text.

Circularity Check

0 steps flagged

No significant circularity: MENTOR's central claims are tested on held-out benchmarks, and the entropy-gating mechanism is a design choice rather than a fitted prediction.

full rationale

The main derivation chain is not circular. MENTOR's core mechanism, Eq. (6), defines a token-level mixture with wt = min(1, Ht/γp); this is an explicit algorithm design, not a quantity fitted to the benchmark results. All headline improvements (MATH, AIME24/25, AMC, GPQA, MMLU-Pro, pass@32) are measured on held-out or out-of-domain evaluations, so the central empirical claim does not reduce to the method's own definitions. The Maximum Entropy argument in Appendix A.1 is a derivation gap: it proves support contraction under a MaxEnt postulate (Jaynes, 1957) rather than from the paper's fitted quantities, but this is an unproven equivalence, not circularity. The assumption that high-entropy tokens identify 'critical decision points' is asserted rather than directly validated, and the claim that guidance is 'restricted to only a few positions' is imprecise because wt is positive at essentially every token. These are correctness/interpretation concerns, not circular reasoning. The only potentially self-referential citation, Du et al. (2024), is used merely as an example of a domain-adapted expert model and is not load-bearing for any result. Accordingly, no circular step can be exhibited with a quote; score 2 reflects the minor, non-load-bearing self-citation and the MaxEnt-related derivation gap, not a reduction of the results to the paper's inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

MENTOR introduces no new entities; it contributes a new training procedure. Its free parameters are hyperparameters of the procedure. The key domain assumption, that entropy selects decision points, is the load-bearing premise on which the method rests.

free parameters (5)
  • quantile p for entropy threshold gamma_p = 0.95 (95th percentile)
    Chosen by hand in Appendix B; determines which tokens receive expert guidance. No ablation is provided.
  • alpha (mixed-policy advantage weight) = 1 annealed to 0 via cosine over 120 steps
    Scheduled decay is a hand-picked schedule to transition from expert-guided to self-driven exploration.
  • number of mixed-policy rollouts N2 = 4 (vs 8 on-policy rollouts)
    Hand-set ratio of mixed to on-policy rollouts; no sensitivity analysis is reported.
  • initial gamma_p at first step = 999
    Special-case value to avoid undefined entropy ratios at the very first training step.
  • reward weights (outcome : format) = 9:1 for Qwen, 8:2 for LLaMA
    Adjusted per model family to enforce output format; not a central theoretical choice.
axioms (3)
  • domain assumption High-entropy tokens in the on-policy distribution correspond to critical decision points where expert guidance is needed.
    Central to MENTOR's design (Eq. 6). No evidence is given that entropy correlates with reasoning correctness or branching; this is assumed.
  • ad hoc to paper Maximum Entropy Principle applies to RLVR optimization.
    Introduced in Appendix A.1 to prove support contraction; not derived from the GRPO objective, and the paper sets KL and entropy regularization to 0.
  • domain assumption The expert model's token distribution is beneficial at high-entropy tokens.
    Assumes a stronger reference model's distribution is helpful at uncertain points; if the expert is wrong at those positions, mixed rollouts could mislead the student.

pith-pipeline@v1.3.0-alltime-deepseek · 17592 in / 9648 out tokens · 79095 ms · 2026-08-04T11:29:57.168505+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Figures

Figures reproduced from arXiv: 2510.04140 by Fei Yu, Jiaqing Liang, Jinyi Han, Shuguang Ma, Sihang Jiang, Tingyun Li, Xinyi Wang, Yanghua Xiao, Zhaoqian Dai, Zishang Jiang.

Figure 1
Figure 1. Figure 1: Illustration of MENTOR framework. By providing expert guidance only at critical decision points, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics of MENTOR compared with On-policy RL. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The occurrence rate of high-frequency reasoning tokens under different training methods. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pass@32 performance of Qwen2.5-7B under different methods. MENTOR improves the model’s reasoning diversity beyond other baselines. To further quantify the impact of different meth￾ods on reasoning diversity, we adopt pass@k as the evaluation metric, which is widely used to measure reasoning diversity (Song et al., 2025; Chen et al., 2025). As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Experience Augmented Policy Optimization for LLM Reasoning

    cs.LG 2026-06 conditional novelty 6.0

    EAPO injects a prior RL policy's token choices at critical decision points during rollout and beats standard RLVR baselines on math and science reasoning benchmarks.

  2. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.5

    Injecting correct reference-policy samples and optimizing a mixed importance-sampling ratio prevents validation collapse and raises asymptotic math-reasoning scores beyond reverse-KL baselines.

  3. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.0

    ARMOR adds correct reference-policy anchor samples to each RL batch and optimizes a reference-mixture importance ratio, preventing validation collapse and extending performance gains.

  4. Experience Augmented Policy Optimization for LLM Reasoning

    cs.LG 2026-06 unverdicted novelty 5.0

    EAPO reuses prior RL policy experience adaptively at decision points in LLM rollouts with adapted importance sampling and reports gains over prior RLVR methods on math benchmarks.

Reference graph

Works this paper leans on

21 extracted references · 17 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  2. [5]

    Does rein- forcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does rein- forcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Pith/arXiv arXiv

  3. [6]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  4. [7]

    Outcome-based exploration for llm reasoning.arXiv preprint arXiv:2509.06941,

    Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for llm reasoning.arXiv preprint arXiv:2509.06941,

    Pith/arXiv arXiv

  5. [8]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Pith/arXiv arXiv

  6. [9]

    Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,

    Pith/arXiv arXiv

  7. [10]

    On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting.arXiv preprint arXiv:2508.11408, 2025a

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jin- gren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting.arXiv preprint arXiv:2508.11408, 2025a. Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. ...

    arXiv

  8. [11]

    Questa: Expanding reasoning capacity in llms via question augmentation.arXiv preprint arXiv:2507.13266,

    Jiazheng Li, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Hongzhou Lin, Yi Wu, and Jingzhao Zhang. Questa: Expanding reasoning capacity in llms via question augmentation.arXiv preprint arXiv:2507.13266,

    arXiv

  9. [12]

    Blending supervised and reinforcement fine-tuning with prefix sampling.arXiv preprint arXiv:2507.01679,

    Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling.arXiv preprint arXiv:2507.01679,

    Pith/arXiv arXiv

  10. [13]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforce- ment learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforce- ment learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Pith/arXiv arXiv

  11. [14]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  12. [15]

    Mitigating strategy-selection bias in reasoning for more effective test-time scaling.arXiv preprint arXiv:2509.17905,

    Zongqian Wu, Baoduo Xu, Tianyu Li, Zhu Sun, Xiaofeng Zhu, and Lei Feng. Mitigating strategy-selection bias in reasoning for more effective test-time scaling.arXiv preprint arXiv:2509.17905,

    arXiv

  13. [16]

    Think thrice before you act: Progressive thought refinement in large language models.arXiv preprint arXiv:2410.13413,

    Chengyu Du, Jinyi Han, Yizhou Ying, Aili Chen, Qianyu He, Haokun Zhao, Sirui Xia, Haoran Guo, Jiaqing Liang, Zulong Chen, et al. Think thrice before you act: Progressive thought refinement in large language models.arXiv preprint arXiv:2410.13413,

    Pith/arXiv arXiv

  14. [17]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    10 A PREPRINT Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    Pith/arXiv arXiv

  15. [18]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling. Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively bala...

    Pith/arXiv arXiv

  16. [19]

    Federico Pennino, Bianca Raimondi, Massimo Rondelli, Andrea Gurioli, and Maurizio Gabbrielli

    URLhttps://openreview.net/ forum?id=PXD3FAVHJT. Federico Pennino, Bianca Raimondi, Massimo Rondelli, Andrea Gurioli, and Maurizio Gabbrielli. From reasoning to code: Grpo optimization for underrepresented languages.arXiv preprint arXiv:2506.11027,

    Pith/arXiv arXiv

  17. [20]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  18. [2022]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Cand `es, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    Pith/arXiv arXiv

  19. [2023]

    ## Hint: Partial Solution

    Appendix A Theoretical Proof A.1 Support Contraction toT ⋆ We now provide a short proof that, under the reward-maximization objective in Eq.(2), the optimal distribu- tion places all probability mass onT ⋆. 11 A PREPRINT For a fixed questionq, writeR(τ)≡R(q, τ)on the discrete setS q = supp(πθ(· |q)). LetR max = supτ∈S q R(τ)and denote the set of maximizer...

    1957

  20. [2024]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  21. [2025]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    Pith/arXiv arXiv