Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

A new RLVR method, EAPO, reuses a prior RL-optimized policy as an action-level experience prior, resampling tokens at critical decision points to improve LLM reasoning over state-of-the-art baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

EAPO injects a prior RL policy's token choices at critical decision points during rollout and beats standard RLVR baselines on math and science reasoning benchmarks.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection EAPO is a plausible empirical method with consistent gains, but its 'stable and unbiased learning' claim does not survive contact with the actual estimator. the 4 major comments →

arxiv 2606.30420 v2 pith:25R6PVE3 submitted 2026-06-29 cs.LG

Experience Augmented Policy Optimization for LLM Reasoning

Jinda Lu , Kexin Huang , Junkang Wu , Shuo Yang , Jinghan Li , Chiyu Ma , Shaohang Wei , Xiang Wang
show 2 more authors
Guoyin Wang Jingren Zhou
This is my paper
classification cs.LG
keywords Reinforcement Learning with Verifiable RewardsLLM reasoningexperience reusepolicy optimizationimportance samplingcritical token resamplingpolicy mismatchQwen
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that experience in reinforcement learning with verifiable rewards should be reused not as fixed reasoning trajectories but as an action-level prior from a previously RL-optimized policy. EAPO identifies critical decision points where the current policy is overconfident but disagrees with the prior policy, resamples tokens from the prior policy only at those points, and corrects the resulting off-policy gradients with a smoothed importance sampling scheme. On five benchmarks across two model families, EAPO consistently outperforms GRPO, DAPO, on-policy distillation, multi-teacher distillation, and trajectory replay. A sympathetic reader would care because the method gets these gains with sparse intervention — fewer than 4% of tokens resampled — and transfers to out-of-domain science reasoning.

Core claim

EAPO establishes that selectively injecting experience from a prior RL-optimized policy at critical decision points improves LLM reasoning over state-of-the-art RLVR methods. At each decoding step, it computes a token-level log-likelihood discrepancy \delta_t = \log \pi_\theta(y_t | q, y_{<t}) - \log \pi_{RL}(y_t | q, y_{<t}); when \delta_t exceeds a threshold \tau, the token is resampled from \pi_{RL}. Training uses a smoothed response-level importance ratio to keep gradients stable, along with positive-only filtering of experience-augmented trajectories and annealing that disables resampling after 60 steps. On Qwen-2.5-Math-7B and Qwen-3-8B-Base, EAPO reaches the highest Pass@1 and Pass@16

What carries the argument

The central mechanism is the gated token-resampling rule: a token-level discrepancy \delta_t compares current and prior policy log-likelihoods, and a gate g_t = I(\delta_t > \tau) triggers resampling from the prior RL policy \pi_{RL}. To keep optimization stable, EAPO uses a response-level smoothed importance ratio \tilde r_i = \pi_\theta(o_i) / ((1-\rho_i)\pi_{old} + \rho_i \pi_{RL}), where \rho_i is the fraction of resampled tokens; when \rho_i = 0 the ratio reduces to standard on-policy importance sampling. Positive experience filtering and block-wise verification (inspired by speculative decoding) complete the machinery.

Load-bearing premise

The importance ratio in Eq. (12) treats the effective rollout policy as a fixed mixture of \pi_{old} and \pi_{RL} weighted by the resampling fraction; if the true behavior policy under gated resampling diverges from this surrogate, the gradient estimate is biased.

What would settle it

On a small held-out set, compute the exact per-token sampling distribution under EAPO's gated resampling (including the gating decisions) and compare the true trajectory probability with \tilde\pi_{Exp}; if the divergence is substantial, the smoothing assumption fails and the unbiasedness claim is contradicted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • EAPO can be applied on top of any RLVR base algorithm (e.g., DAPO) and yields consistent gains across model scales and benchmarks, suggesting experience reuse is a general orthogonal improvement.
  • Because resampling is sparse (resampled ratio below 4%), the method significantly cuts exploration cost while maintaining on-policy exploration.
  • The improvement on out-of-distribution science benchmarks (GPQA, MMLU-Pro) indicates EAPO learns transferable reasoning behaviors rather than memorizing training patterns.
  • EAPO with smoothed importance sampling outperforms both trajectory-level replay and token-level supervision methods, implying that critical-token injection is a more effective form of experience reuse.
  • The smoothing and positive filtering components are complementary; ablations show monotonic gains when added incrementally.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • EAPO's critical-token detection could be used as a diagnostic to identify exactly which reasoning steps the current policy is unsure about, potentially enabling human-understandable explanations of policy failure.
  • The response-level smoothed importance ratio is a heuristic approximation; if it is replaced with a per-token causal importance correction, the method might achieve unbiased gradients and further gains — a testable variant.
  • The dependence on a single fixed \pi_{RL} is a limitation acknowledged in the paper; extending to multiple or evolving experience policies may improve robustness and is a natural next step.
  • The threshold \tau and annealing step T are fixed; adapting them to training dynamics could make the method less sensitive, which the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Experience-Augmented Policy Optimization (EAPO), a method for LLM reasoning with verifiable rewards. EAPO uses a previously RL-optimized policy πRL as an action-level prior: at decoding steps where the current policy is overconfident relative to πRL (measured by a log-likelihood discrepancy δ_t > τ), the token is resampled from πRL. A block-wise variant amortizes the discrepancy check. For optimization, the paper introduces positive experience filtering (keeping only successful experience-augmented trajectories) and a smoothed importance-sampling ratio based on the fraction ρ of resampled tokens. The method is evaluated on AIME'24, AIME'25, AMC, GPQA, and MMLU-Pro with Qwen-2.5-Math-7B and Qwen-3-8B, reporting consistent improvements over GRPO, DAPO, on-policy distillation, multi-teacher distillation, and trajectory replay. The central theoretical claim is that the smoothed importance-sampling scheme yields 'stable and unbiased learning' from experience-augmented rollouts.

Significance. The empirical direction is interesting and timely: reusing an RL-optimized policy as a sparse, action-level prior is a more policy-adaptive alternative to trajectory replay, and the reported gains are consistent across five benchmarks and two base models. The ablations on resampling threshold, experience granularity, and positive filtering provide useful evidence for the design choices. However, the paper's theoretical contribution as stated—unbiased importance-sampling correction for the gated sampling process—is not sound. If the authors reframe the smoothed ratio as a heuristic correction or provide a valid unbiased estimator, the empirical results would still be worth publishing. The current version overclaims statistical guarantees that are not delivered, so substantial revision is needed before the paper can be accepted.

major comments (4)
  1. [§3.3, Eqs. (10)–(12)] The abstract and Sec. 3.3 state that the adapted importance-sampling scheme ensures 'stable and unbiased learning.' This is not established and is false as stated. The actual rollout behavior in Eqs. (7)–(8) is a per-token thresholded mixture: q_t(a)=π_old(a)1{δ_t(a)≤τ}+α_t π_RL(a), with α_t=Σ_{a:δ_t(a)>τ}π_old(a). The surrogate \tilde{π}^b_Exp=(1−ρ)π_old+ρπ_RL is a response-level mixture with ρ the realized fraction of resampled tokens; it coincides with the true behavior only when ρ=0 or ρ=1. For 0<α_t<1 the two differ, and no error bound is supplied. The unbiasedness claim should be removed or replaced by a corrected estimator/proof.
  2. [§3.3, Eq. (9) and Algorithm 1 lines 28–31] Positive filtering discards experience-augmented trajectories with R=0 from the gradient. This conditions the update on R=1 without an importance-weight or control-variate correction. The effective objective is an expectation under q_Exp(·|R=1), not under q_Exp; even if the surrogate Eq. (11) were exact, the estimator would be biased for the stated objective. If positive filtering is intended as a heuristic variance-reduction device, it should be labeled as such rather than as part of an unbiased scheme.
  3. [§4.2, Figure 2 and Appendix A] The threshold τ is selected using AIME'24 performance ('we adopt τ=0.5 as the default setting' from Figure 2a), and Figure 2a reports 'the best performance achieved during training.' Likewise T=60 is chosen from AIME'24 in Figure 4a. The headline AIME'24 comparison in Table 1 is therefore a selected statistic from tuning on the test benchmark, not an independent prediction. Re-report final-checkpoint results or use a separate validation set; the other benchmarks (AIME'25, AMC, GPQA, MMLU-Pro) are less affected but should also use a fixed protocol.
  4. [§3.3, 'Resampling-based Importance Sampling'] The paper claims token-level correction 'breaks their causal dependency.' This is incorrect: autoregressive importance-sampling ratios factorize into per-token conditional ratios; resampling a prefix changes the conditioning, which is exactly what the per-token product accounts for. The response-level surrogate in Eq. (11) is not a valid behavior policy for the gated process. If token-level correction is avoided for computational or stability reasons, Eqs. (10)–(12) should be presented as a heuristic, not as an unbiased estimator.
minor comments (6)
  1. [Table 1] The Trajectory Replay row for Qwen-2.5-Math-7B reads '72.7489.9342.08' with missing separators; fix formatting.
  2. [Tables 1 and 2] Results are averages over 32 or 8 runs but no standard deviations or confidence intervals are reported. Given the small margins on some benchmarks (e.g., MMLU-Pro on Qwen-3-8B), error bars are needed to support the 'consistent improvement' claim.
  3. [§4.1] The training budget for obtaining πRL is not specified (number of steps, convergence criteria, compute). Without this, the two-stage procedure is hard to reproduce.
  4. [§4.3] The ablation 'EAPO w/o sIS' is not fully specified: what importance ratio replaces Eq. (12) in that variant? This is needed to interpret the ablation.
  5. [Eq. (12)] The notation \tilde{π}^b_Exp(o_i^b|q^b) suggests a mixture of full-sequence distributions, but the denominator is never defined explicitly as a product over tokens. Clarify the exact probability model.
  6. [Figure 2] The metric 'Mean@32' is used in the main text and figures but never defined. Define it precisely in Sec. 4.1.

Circularity Check

1 steps flagged

EAPO's headline AIME'24 result is partly fitted via hyperparameters selected on AIME'24; the derivation itself is not circular and other benchmarks are independent.

specific steps
  1. fitted input called prediction [Section 4.2 (Hyperparameter Sensitivity, RQ1), Figure 2 caption; Appendix A (Effect of experience annealing)]
    "To minimize reliance on πRL while maintaining strong performance, we adopt τ=0.5 as the default setting. This configuration achieves a favorable balance between sparse experience usage and effectiveness, reaching a Mean@32 score of 0.381 on AIME’24. ... Figure 2. Sensitivity analysis of the resampling threshold τ. (a) AIME’24 performance (Mean@32) under different values of τ, reporting the best performance achieved during training."

    The default resampling threshold τ and the annealing step T are selected by inspecting AIME'24 performance, and Figure 2(a) reports the best AIME'24 score over training. The main Table 1 results for AIME'24 therefore partly reflect hyperparameters tuned on that exact benchmark rather than an out-of-sample prediction. However, AIME'25, AMC, GPQA, and MMLU-Pro are not used for this selection, and EAPO's core mechanism is not derived from these numbers, so the circularity is partial and confined to the AIME'24 headline.

full rationale

The core EAPO derivation is self-contained: the resampling gate (Eq. 7–8), the smoothed importance ratio (Eq. 10–12), and the DAPO-based objective are defined from first principles in the paper, and the method's performance is evaluated against external baselines and multiple held-out benchmarks. The self-citations to Huang et al. (2026) and other own-team papers provide background motivation about pivotal tokens but are not the load-bearing proof of EAPO's effectiveness; independent citations (Wang et al., 2025; Cheng et al., 2025) support the same premise. The unsupported unbiasedness claim for the smoothed importance-sampling surrogate (Eq. 11) is a correctness/rigor concern, not a circularity, since the surrogate is introduced as an approximation rather than derived from the target. The only genuine circular step is the hyperparameter selection: τ and T are tuned on AIME'24 and the best AIME'24 training score is used as evidence, making that specific benchmark partly a fitted result. The other four benchmarks are untouched by this selection and give independent grounding, so the central claim is not forced by definition.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The method depends on the prior policy πRL and on unproven statistical assumptions: the smoothed IS surrogate (Eq. 11) and the validity of positive filtering within group-normalized advantage estimation (Eq. 4/9). Three hyperparameters (τ, T, K) are hand-set/tuned on AIME'24. The binary reward and DAPO objective are standard inputs, not new entities.

free parameters (4)
  • resampling threshold τ = 0.5
    Controls gating δ_t>τ; chosen from AIME'24 sensitivity (Fig. 2), not derived.
  • annealing step T = 60
    Controls how many training steps use experience; tuned on AIME'24 (Fig. 4a).
  • block size K = 20
    Number of speculative tokens checked per verification step; chosen for efficiency, not ablated.
  • experience-to-on-policy ratio in group = 1 exp : 15 on-policy
    Group composition G=16 with one experience-augmented trajectory; design choice not ablated.
axioms (6)
  • domain assumption Binary verifiable reward R(·) returns 1 iff extracted answer equals ground truth.
    Standard RLVR setup, Eq. (1); the whole method inherits the accuracy of this reward signal.
  • domain assumption Prior RL policy πRL trained by DAPO on DAPO-Math-17K is a reliable action-level experience prior.
    Section 3.1/4.1; effectiveness is bounded by πRL quality, acknowledged in Limitations.
  • domain assumption Token-level log-likelihood ratio δ_t = log πθ - log πRL identifies 'critical decision points' where expert intervention helps.
    Section 3.2, Eq. (6)-(7); this is a heuristic premise validated only indirectly by ablation.
  • ad hoc to paper The smoothed mixture \tildeπExp = (1-ρ)πold + ρπRL adequately approximates the true gated per-token behavior policy for importance sampling.
    Section 3.3, Eq. (11); no proof or error bound is given; actual behavior policy is per-token gated and trajectory-dependent, so this surrogate is generally not the true behavior policy.
  • ad hoc to paper Group-relative advantage normalization remains valid when failed experience-augmented trajectories are filtered out.
    Section 3.3 positive filtering + Eq. (4); excluding failed trajectories changes the group mean/std and conditions the estimator on success, with no correction.
  • standard math Block-wise verification with block size K preserves the original resampling behavior.
    Section 3.2; if the earliest critical index is resampled and later speculative tokens discarded, behavior matches per-token verification, assuming gating evaluated on the same prefixes.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Experience Augmented Policy Optimization for LLM Reasoning." pith.science (2026). https://pith.science/paper/25R6PVE3

@misc{pith2026260630420,
  author       = {Pith},
  title        = {Pith review of: Experience Augmented Policy Optimization for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25R6PVE3}},
  note         = {Machine review of arXiv:2606.30420}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.

Figures

Figures reproduced from arXiv: 2606.30420 by Chiyu Ma, Guoyin Wang, Jinda Lu, Jinghan Li, Jingren Zhou, Junkang Wu, Kexin Huang, Shaohang Wei, Shuo Yang, Xiang Wang.

Figure 1
Figure 1. Figure 1: Overview of experience-augmented reinforcement learning. (a) Standard reinforcement learning with verifiable rewards (RLVR) optimizes a base policy πBase into an RL-optimized policy πRL through iterative rollouts and policy updates. (b) Instead of training from scratch, EAPO leverages experience from a prior RL-optimized policy πRL during rollout of the current policy by resampling actions at critical deci… view at source ↗
Figure 2
Figure 2. Figure 2: Sensitivity analysis of the resampling threshold τ . (a) AIME’24 performance (Mean@32) under different values of τ , reporting the best performance achieved during training. (b) Evolution of the resampled ratio across training steps, representing the proportion of resampled tokens within the experience-augmented responses. (c) Accuracy of experience-augmented responses across training steps, reflecting the… view at source ↗
Figure 3
Figure 3. Figure 3: Influence of experience granularity and filtering strategy. (a) AIME’24 performance (Mean@32) comparison between token-level (Tokens, Pos-Only) and trajectory-level (Trajectory, Pos-Only) experience. Following (Zhan et al., 2025; Liu et al., 2025), we consider only positive samples in this comparison, where positive samples refer to trajectories whose final predictions are correct. (b) KL divergence to the… view at source ↗
Figure 4
Figure 4. Figure 4: Additional ablation results. Left: impact of experience annealing steps. Right: complementary effects of EAPO components. Effect of experience annealing. Figure 4a studies the impact of the experience annealing steps T. Across all settings, incorporating experience consistently improves performance over the baseline, confirming the general benefit of experience￾guided resampling. When the annealing period … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.5

    Injecting correct reference-policy samples and optimizing a mixed importance-sampling ratio prevents validation collapse and raises asymptotic math-reasoning scores beyond reverse-KL baselines.

  2. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.0

    ARMOR adds correct reference-policy anchor samples to each RL batch and optimizes a reference-mixture importance ratio, preventing validation collapse and extending performance gains.

Reference graph

Works this paper leans on

19 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    Pith/arXiv arXiv

  2. [5]

    Leviathan, Y ., Kalman, M., and Matias, Y

    URL https://arxiv.org/ abs/2510.04140. Leviathan, Y ., Kalman, M., and Matias, Y . Fast inference from transformers via speculative decoding. InInter- national Conference on Machine Learning, pp. 19274– 19286. PMLR,

    Pith/arXiv arXiv

  3. [6]

    Li, S., Zhou, Z., Lam, W., Yang, C., and Lu, C

    URL https://openreview.net/forum?id= jyOgpu5wfC. Li, S., Zhou, Z., Lam, W., Yang, C., and Lu, C. Repo: Replay-enhanced policy optimization.arXiv preprint arXiv:2506.09340, 2025b. Liang, J., Tang, H., Ma, Y ., Liu, J., Zheng, Y ., Hu, S., Bai, L., and Hao, J. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model....

    Pith/arXiv arXiv

  4. [7]

    Deepseek-v3

    Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556,

    Pith/arXiv arXiv

  5. [8]

    Rethinking visual content refinement in low-shot clip adaptation.arXiv preprint arXiv:2407.14117,

    Lu, J., Wang, S., Hao, Y ., Liu, H., Wang, X., and Wang, M. Rethinking visual content refinement in low-shot clip adaptation.arXiv preprint arXiv:2407.14117,

    Pith/arXiv arXiv

  6. [10]

    https://thinkingmachines.ai/blog/on-policy- distillation

    doi: 10.64434/tml .20251026. https://thinkingmachines.ai/blog/on-policy- distillation. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling,

    doi:10.64434/tml

  7. [11]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  8. [13]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534,

    Team, K., Bai, Y ., Bao, Y ., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y ., Chen, Y ., Chen, Y ., et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534,

    Pith/arXiv arXiv

  9. [14]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Pith/arXiv arXiv

  10. [15]

    Quantile advantage estimation for entropy-safe reasoning

    Wu, J., Huang, K., Wu, J., Zhang, A., Wang, X., and He, X. Quantile advantage estimation for entropy-safe reasoning. arXiv preprint arXiv:2509.22611,

    arXiv

  11. [16]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al

    URL ht tps://openreview.net/forum?id=vO8LLo NWWk. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024a. Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward...

    Pith/arXiv arXiv

  12. [17]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  13. [18]

    F., and Cheng, Y

    Zhan, R., Li, Y ., Wang, Z., Qu, X., Liu, D., Shao, J., Wong, D. F., and Cheng, Y . Exgrpo: Learning to reason from experience.arXiv preprint arXiv:2510.02245,

    arXiv

  14. [19]

    Rlep: Reinforcement learning with experience replay for llm reasoning.arXiv preprint arXiv:2507.07451, 2025a

    Zhang, H., Fu, J., Zhang, J., Fu, K., Wang, Q., Zhang, F., and Zhou, G. Rlep: Reinforcement learning with experience replay for llm reasoning.arXiv preprint arXiv:2507.07451, 2025a. Zhang, K., Chen, X., Liu, B., Xue, T., Liao, Z., Liu, Z., Wang, X., Ning, Y ., Chen, Z., Fu, X., et al. Agent learning via early experience.arXiv preprint arXiv:2510.08558, 20...

    Pith/arXiv arXiv

  15. [2017]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  16. [2023]

    X., Zhang, Z., and Wei, F

    Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, W. X., Zhang, Z., and Wei, F. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Pith/arXiv arXiv

  17. [2024]

    Adavip: Aligning multi-modal llms via adaptive vision- enhanced preference optimization, 2025a

    Lu, J., Li, J., Gao, Y ., Wu, J., Wu, J., Wang, X., and He, X. Adavip: Aligning multi-modal llms via adaptive vision- enhanced preference optimization, 2025a. URL https: //arxiv.org/abs/2504.15619. Lu, J., Wu, J., Li, J., Jia, X., Wang, S., Zhang, Y ., Fang, J., Wang, X., and He, X. Dama: Data-and model-aware alignment of multi-modal llms. InInternational...

    Pith/arXiv arXiv

  18. [2025]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  19. [2026]

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al

    URL https://openreview.net/forum ?id=r6Pw3RiMYL. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.