Pith. sign in

REVIEW 3 major objections 5 minor 59 references

This paper claims that a game solver's one-step cost-to-go change supplies cheap turn-level credit for LLM agents, and that maximizing this solver advantage is equivalent to logit-free on-policy distillation, yielding consistent gains over

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:51 UTC pith:UKYKHLTE

load-bearing objection Strong empirical method, over-sold theory: the solver-advantage signal works, but the 'logit-free distillation' claim reduces to reward shaping once you look at the solver being used. the 3 major comments →

arxiv 2607.25308 v1 pith:UKYKHLTE submitted 2026-07-28 cs.CL cs.AI

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

Yu Wang , Yi-Kai Zhang , Wentao Shi , Ziang Ye , Yuchun Miao , Yueqing Sun , Qi Gu , Xunliang Cai
show 3 more authors
Lan-Zhe Guo Han-Jia Ye Fuli Feng
This is my paper
classification cs.CL cs.AI
keywords credit assignmentRLVRon-policy distillationsolver advantageprocess supervisionLLM agentsgame solverszero-shot transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon game agents trained with reinforcement learning from verifiable rewards (RLVR) face a credit-assignment problem: a terminal win/loss signal does not say which actions mattered. CAST proposes to query a game solver on each state the LLM visits, define the solver advantage as the one-step drop in the solver's cost-to-go, and add that scalar as dense turn-level credit alongside the trajectory reward. The paper's theoretical claim is that under a soft-optimal solver assumption this update is equivalent to on-policy distillation from the solver without teacher logits, because the scalar advantage already encodes the teacher's log-preference. Empirically, CAST outperforms trained baselines on Sokoban, Minesweeper, and Rush Hour under both in-domain and unseen-difficulty evaluation and attains the best zero-shot averages on ALFWorld and WebShop. If correct, the result means process supervision for LLM agents can come from existing solvers or learned value functions at negligible training overhead.

Core claim

The central discovery is that the one-step drop in a solver's cost-to-go, N(s_t) − N(s_{t+1}), is a usable turn-level credit signal. The paper proves that, under a soft-optimal solver with small advantages, the update maximizing this shifted solver advantage equals the gradient of task return minus a cross-entropy to the solver. This is logit-free distillation: a single scalar carries the teacher's log-preference, so no logits are needed. The closed-form optimal policy tilts the solver distribution by the task advantage, letting the student surpass the solver. Empirically, CAST gets the best Avg@4 on every game in both settings, reaches DAPO's peak in 1.7–2.0x fewer steps, and transfers zero

What carries the argument

The central object is the shifted solver advantage, eA(s_t,a_t) = N(s_t) − N(s_{t+1}), where N(s) is the solver's cost-to-go: how much work remains to win. Each LLM-sampled action is scored this way, then passed through asinh compression and batch RMS normalization to tame extremes and align scales across games. The load-bearing identity A = τ log π_solver(a|s) converts the scalar advantage into the teacher's log-preference, making the per-step signal a cross-entropy distillation penalty in the implicit objective while the GRPO outcome advantage anchors task return.

Load-bearing premise

The load-bearing premise is that the solver acts as a soft-optimal teacher with a positive temperature, so its advantage equals a scaled log-probability; the deployed deterministic near-optimal solvers sit at the hard-max limit, and the paper concedes the distillation interpretation degrades proportionally.

What would settle it

Record the distribution of one-step cost-to-go changes A = N(s_t) − N(s_{t+1}) over a CAST training run. If a substantial share of steps have |A| > 1 — for instance, transitions into dead states with penalty −N(s_t) — Assumption C.2 fails and the constant-beta distillation equivalence of Theorem 2.1 is quantitatively inaccurate; the method would then have to be explained by the robustified step-dependent beta or by reward shaping rather than by logit-free distillation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, long-horizon LLM agents can be trained with dense turn-level credit using only scalar solver values — no teacher logits, learned reward models, or expensive search.
  • Training with the solver advantage reaches the outcome-only backbone's peak validation performance in 1.7–2.0x fewer steps and continues to a higher plateau on Sokoban, Minesweeper, and Rush Hour.
  • The benefit transfers zero-shot: game-trained CAST agents beat all trained baselines on ALFWorld and WebShop, suggesting general planning abilities rather than game-specific memorization.
  • A learned value network can replace the exact solver and retain most of the gain, so the mechanism should apply wherever a reliable state-value estimate exists.
  • Because the implicit objective is cross-entropy plus a task-return tilt, the student is not restricted to imitating the solver; in the closed-form solution the task advantage lets the student exceed the teacher.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The practical mechanism may be more modest than the headline: for the deterministic near-optimal solvers actually deployed, the paper's own Proposition C.12 identifies the update as potential-based reward shaping, and the distillation interpretation degrades proportionally. The theorem likely explains why shaping helps rather than proving the deployed algorithm distills.
  • A natural extension is to test whether the method works in non-game agentic benchmarks where no exact solver exists but a learned value model can be trained; the DQN variant on Rush Hour suggests the bottleneck is value quality, not solver exactness.
  • The zero-shot transfer results invite a controlled study of what game-trained agents actually learn — e.g., whether the WebShop gains come from a general 'check constraints before committing' behavior rather than from game-specific action patterns.
  • The paper's robustified-coefficient corollary implies that the asinh transformation is not just a stabilizer but an adaptive distillation-temperature schedule; one could test this by measuring beta_eff across training and correlating it with performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. CAST (Credit Assignment from Solver Teachers) augments GRPO/DAPO policy-gradient training of LLM game agents with a turn-level scalar signal derived from a game solver's state-value change: for each sampled action the agent compares the solver's cost-to-go before and after, applies asinh compression and batch RMS normalization, and adds the result to the group-relative outcome advantage (Eq. 8). The paper claims (Theorem 2.1, Eq. 9) that, under a soft-optimal-solver assumption, this update is equivalent to on-policy distillation from the solver using only scalar values, with no teacher logits. Experiments on Qwen3-4B-Instruct-2507 across Sokoban, Minesweeper, and Rush Hour report the best Avg@4 among trained baselines on all three games for both in-domain and unseen difficulty (ID average 62.1 vs 44.7 for DAPO), faster attainment of DAPO's peak, and the highest zero-shot averages on ALFWorld (37.9) and WebShop (22.7). Ablations examine the weight, the asinh transformation, RMS normalization, a learned value network as solver, and overhead.

Significance. The empirical comparison is well controlled: same base model, same sparse 0/1 reward, same rollout budget, and a clean DAPO-backbone ablation; the gains are consistent across games and transfer. The paper ships code and states its assumptions explicitly, which is commendable. If the theoretical equivalence held, the logit-free distillation result would be a significant contribution to RLVR. However, as detailed below, the theorem's key assumption is not satisfied by the deployed deterministic solvers, and the actual mechanism is potential-based reward shaping (Ng et al., 1999), which is standard and policy-invariant but not the claimed distillation novelty. The empirical contribution remains publishable if the theoretical claims are reframed honestly.

major comments (3)
  1. [§2.2 Theorem 2.1 / App. C.5, Assumption C.1] Assumption C.1 (Eq. 14) requires the solver to be a soft-optimal policy with finite temperature τ>0, which yields the identity A^{π_Solver}=τ log π_Solver (Lemma C.5). The three deployed solvers are deterministic optimal/near-optimal planners (Appendix C.3: weighted A*, multi-source reverse BFS, peek-free CSP search). A deterministic policy cannot be expressed as exp(Q/τ)/Z with finite Q-values and τ>0; as τ→0, the right side of Lemma C.5 diverges for any suboptimal action (approximately -Δ/τ), while the implemented update α h(Ã_{i,t}) in Eq. 8 is O(1) and contains no τ. Hence the coefficient β in Eq. 9 is not identifiable from the implemented update, and the 'logit-free on-policy distillation' interpretation does not apply to the actual experiments. The paper itself concedes this in (A1) of Appendix C.5.7 and then identifies the real mechanism as potential-based reward shaping (Prop. C.
  2. [App. C.5.2, Assumption C.3 / Eq. 2] Assumption C.3 states that GRPO's group-relative return (Eq. 2) is an unbiased estimator of the task advantage up to a positive scaling constant. This is asserted, not proved. With G=8 rollouts per prompt and 0/1 sparse returns, any group in which all or none of the trajectories succeed has σ_R=0, so every member receives advantage 0 regardless of which actions were taken; the resulting gradient estimate is zero even when the true policy gradient is nonzero. Thus Eq. 2 is not an unbiased estimator of A^{π_θ}_{task} in finite samples. Step 2 of the proof of Theorem 2.1 depends on this identification. Please either provide a rigorous finite-sample statement (with the exact conditions under which unbiasedness holds), or weaken the theorem to an approximation and validate the approximation empirically (e.g., by comparing to a learned critic or Monte-Carlo advantage estimator).
  3. [§2.2 Eq. 5 / Prop. C.12] Proposition C.12 interprets the solver signal as potential-based reward shaping (Ng et al., 1999) and leverages that theorem's optimal-policy invariance. However, Eq. 5's treatment of dead states is not a potential difference: when N(s_{t+1})=∞, the paper caps the transition at the finite penalty -N(s_t). This cap is not of the form Φ(s_t)-Φ(s_{t+1}) for any well-defined potential Φ, so the shaping invariance guarantee does not literally hold for dead-state transitions. Those transitions are exactly the large-magnitude signals that the asinh compression is designed to dampen (Fig. 4), so the discrepancy is not negligible. The shaping interpretation and the PDL-based bound of Eq. 23 should be stated for the uncapped signal, with a separate analysis (or at least an explicit acknowledgment) for the capped version used in training.
minor comments (5)
  1. [Table 1] Table 1 reports means over 3 runs without standard deviations or significance tests. Given the headline 'best in every game,' please add error bars or confidence intervals.
  2. [§2.2, Eq. 4] Section 2.2 defines N(s) as the minimum work to reach a win, but Minesweeper's solver (Appendix C.3) is a deterministic peek-free effort estimate K(s), not an optimal cost-to-go; this mismatch should be acknowledged where Eq. 4 is introduced.
  3. [Theorem 2.1, Eq. 9] Eq. 9 writes J(θ) as a fixed objective, but β contains the batch-level RMS_B(g), which is data-dependent. The 'equality' in Theorem 2.1 is therefore at best a per-batch surrogate; clarify this in the statement.
  4. [§2.2, 'Why the student can surpass the solver'] The claim that the student can surpass the solver is not tested; there is no comparison between CAST's success rate and the solver's own success rate on the evaluation tiers. If this is a contribution, add such a comparison.
  5. [Front matter / references] Minor text artifacts: ' /envel⌢pe' appears in author affiliations; the citation 'Shridhar et al.' is missing a year and venue; several other references have inconsistent formatting.

Circularity Check

1 steps flagged

Theorem 2.1's logit-free distillation equivalence is the soft-optimality assumption restated (Lemma C.5 = Eq. 14), and the paper concedes the assumption fails for its deployed deterministic solvers.

specific steps
  1. self definitional [Appendix C.5.2 (Assumption C.1, Eq. 14) and Appendix C.5.3 (Lemma C.5, Eq. 15); used in Theorem 2.1 / Eq. 9]
    "Lemma C.5 (Advantage–Log-Probability Identity). Under Assumption C.1, for all (s,a): AπSolver(s,a)=τ log πSolver(a|s). Proof. Taking the logarithm of Equation 14 gives log πSolver(a|s)=1/τ(QπSolver(s,a)−VπSolver(s))=1/τ AπSolver(s,a). Multiplying both sides by τ yields Equation 15."

    Theorem 2.1's central conclusion — that maximizing the shaped solver advantage is logit-free on-policy distillation — rests on Lemma C.5, which is obtained by taking the logarithm of Assumption C.1's definition. Eq. 14 already stipulates πSolver(a|s)=exp((Q−V)/τ), i.e., the solver's log-probability is defined to be proportional to its advantage. Thus 'A=τ log πSolver' is the soft-optimality assumption rewritten, not an independently derived property of the deployed solvers. The paper's own A1 concession ('the distillation interpretation degrades proportionally' for non-soft-optimal solvers) confirms that the headline equivalence is a property of the assumed Boltzmann teacher, not of the implemented deterministic A*/BFS/CSP solvers. The remaining steps of the theorem are algebra and Taylor

full rationale

The only significant circular step is the derivation of the 'logit-free OPD' equivalence: Lemma C.5 follows immediately from the definition in Assumption C.1, so the theorem's load-bearing identity is the assumption restated. The paper is transparent about this and even acknowledges (A1) that the distillation interpretation degrades proportionally when the solver is not soft-optimal; since the three deployed solvers are deterministic planners, the theorem describes a hypothetical Boltzmann teacher rather than the actual signal source. This reduces the theoretical novelty but does not infect the empirical results, which are self-contained, compared against external baselines and held-out benchmarks, and do not involve fitting a parameter and then 'predicting' a forced consequence. There is no load-bearing self-citation chain or imported uniqueness theorem. Overall, the central theoretical claim is partially circular-by-assumption, while the empirical contribution remains independent, giving a moderate score of 4.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The empirical claim leans on informative cost-to-go oracles and stable shaping; the theoretical claim leans on four named assumptions (C.1–C.4), of which the soft-optimal-solver and GRPO-consistency items are idealizations relative to the deployed deterministic solvers and the group-normalized estimator, respectively. No invented entities: the "solver advantage" is a defined scalar and the soft-optimal teacher is an interpretive device, not a new object requiring independent evidence. The method's main free parameters are the solver-advantage weight α (tuned on Sokoban) and the symbolic temperature τ of the theory; the DQN variant adds hand-set deployment constants. Everything else (asinh, RMS normalization, dead-state cap) is a fixed design rule rather than a fitted quantity.

free parameters (3)
  • α (solver-advantage weight) = 0.1 (default)
    Tuned via validation sweep on Sokoban (Fig. 4a); 0.1 best on both Val Avg@4 and training reward; used at 0.1 for all games without per-game tuning.
  • τ (soft-optimal temperature) = unestimated (assumed >0)
    Internal to Assumption C.1; β in Theorem 2.1 depends on it, but the algorithm never estimates it — the theorem's constants are symbolic.
  • DQN-variant constants = scale=2; solved-state value=1.0; geometric-prior weight −0.05
    Hand-set in Appendix C.4 (Table 4) for the learned-value-network analysis on Rush Hour; not swept.
axioms (7)
  • ad hoc to paper Assumption C.1: solver acts as a soft-optimal policy π(a|s) ∝ exp((Q^πSolver(s,a)−V^πSolver(s))/τ) for τ>0 (Eq. 14)
    Carries the entire distillation equivalence; the deployed solvers are deterministic near-optimal (hard-max) planners, so the assumption is an idealization the paper itself qualifies (C.5.7 A1).
  • domain assumption Assumption C.2: small-signal regime |A^πSolver| ≲ 1, so asinh(A+1) linearizes around the optimal-move point A=0
    Holds for typical ±1 cost-to-go changes but is violated by dead-state penalties −N(s); Corollary C.10 extends the theorem with a step-varying βeff.
  • ad hoc to paper Assumption C.3: GRPO group-relative advantage is an unbiased estimator of the task advantage up to positive scaling
    Not literally true: Eq. 2 normalizes by the group standard deviation, a nonlinear transform; the paper asserts unbiasedness of the gradient direction without proof (C.5.7 A3).
  • standard math Assumption C.4: frozen visitation surrogate (∇θ d^πθ term dropped)
    Standard PPO-style approximation (Schulman et al., 2017); acknowledged in C.5.7 A4.
  • standard math Potential-based reward shaping theorem and Performance Difference Lemma
    Cited standard results used in Prop. C.12 and Prop. C.11 (Ng et al. 1999; Kakade & Langford 2002).
  • domain assumption Minesweeper: revealed-board information state is a sufficient statistic, and the deterministic peek-free solver's K(s) is a usable cost-to-go surrogate
    K(s) is not the true optimal cost-to-go under partial observability (the paper says so in Appendix C.3); on boards requiring guesses, the signal can be +∞-capped and heuristic.
  • domain assumption Deterministic transitions in Sokoban and Rush Hour: E[N(s_{t+1})] = N(s_{t+1})
    Explicit in Eq. 4; the paper keeps the expectation "for generality" but uses the deterministic reduction in practice.

pith-pipeline@v1.3.0-alltime-deepseek · 26575 in / 33220 out tokens · 313272 ms · 2026-08-01T02:51:18.443440+00:00 · methodology

0 comments
read the original abstract

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.

Figures

Figures reproduced from arXiv: 2607.25308 by Fuli Feng, Han-Jia Ye, Lan-Zhe Guo, Qi Gu, Wentao Shi, Xunliang Cai, Yi-Kai Zhang, Yuchun Miao, Yueqing Sun, Yu Wang, Ziang Ye.

Figure 1
Figure 1. Figure 1: Overview. (Left) Several current closed-source LLMs struggle on classical games. (Right) Outcome-only RLVR lacks fine-grained feedback, whereas our solver-derived signal provides accurate turn-level credit. as GiGPO (Feng et al., 2026), yet still face trade-offs among computation, supervision, and signal reliability. We observe that game solvers can provide this missing signal, as a solver can evaluate an … view at source ↗
Figure 2
Figure 2. Figure 2: Method overview. We augment GRPO’s outcome advantage with a shifted solver advantage derived from turn-level cost-to-go changes. 2.2 SOLVER-GUIDED TURN-LEVEL CREDIT A per-step score from the solver. To score each action individually, we need a way to measure how good the current state is. The 0/1 terminal reward cannot do this, as it only takes a value at the very end. A solver can, because it completes a … view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics. Horizontal dashed lines mark DAPO’s peak validation Avg@4; vertical dotted lines mark when CAST and DAPO first reach it, in orange and blue respectively. Curves are EMA-smoothed at 0.6, with bands showing the standard deviation over three runs [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation studies on Sokoban. Left column: sweeping the solver-advantage weight α. Right column: removing/replacing the asinh transformation and batch-level RMS normalization. Curves are EMA-smoothed at 0.6. 1. Solver-advantage weight α ( [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Learned value network as a solver on Rush Hour. We replace the exact solver with a DQN-based value network trained without solver distances as the process-signal source. Curves are EMA-smoothed at 0.6; bands show the standard deviation across the 3 runs. Learned value networks as solvers. Solver guidance can also use an approximate learned value function as the process-signal source, which extends it to th… view at source ↗
Figure 6
Figure 6. Figure 6: Solver overhead on Sokoban. Top: wall-clock breakdowns at three granularities. The left pie decomposes an average batch-level optimizer step (190.0 s), which includes the parallel collection of 128 trajectories (16 prompts × 8 rollouts), the policy update, and other operations. Rollout collection, policy update, and other operations account for 60.7%, 15.5%, and 23.7% of step time, respectively; the last c… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 16 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2402.03300 , year =

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author =. arXiv preprint arXiv:2402.03300 , year =

    Pith/arXiv arXiv

  2. [2]

    Advances in Neural Information Processing Systems , volume =

    Dapo: An open-source llm reinforcement learning system at scale , author =. Advances in Neural Information Processing Systems , volume =

  3. [3]

    arXiv preprint arXiv:2507.18071 , year =

    Group sequence policy optimization , author =. arXiv preprint arXiv:2507.18071 , year =

    Pith/arXiv arXiv

  4. [4]

    Advances in Neural Information Processing Systems , volume =

    Group-in-group policy optimization for llm agent training , author =. Advances in Neural Information Processing Systems , volume =

  5. [5]

    International Conference on Learning Representations (ICLR) , year =

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  6. [6]

    Voyager: An Open-Ended Embodied Agent with Large Language Models , author =

  7. [7]

    International Conference on Machine Learning , pages =

    Cradle: Empowering Foundation Agents towards General Computer Control , author =. International Conference on Machine Learning , pages =. 2025 , organization =

    2025

  8. [8]

    Advances in neural information processing systems , volume =

    Reflexion: Language agents with verbal reinforcement learning , author =. Advances in neural information processing systems , volume =

  9. [9]

    Advances in Neural Information Processing Systems , volume =

    Minedojo: Building open-ended embodied agents with internet-scale knowledge , author =. Advances in Neural Information Processing Systems , volume =

  10. [10]

    International Conference on Learning Representations , volume =

    Balrog: Benchmarking agentic llm and vlm reasoning on games , author =. International Conference on Learning Representations , volume =

  11. [11]

    arXiv preprint arXiv:2505.15146 , year =

    lmgame-Bench: How Good are LLMs at Playing Games? , author =. arXiv preprint arXiv:2505.15146 , year =

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2504.11442 , year =

    Textarena , author =. arXiv preprint arXiv:2504.11442 , year =

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2604.07429 , year =

    GameWorld: Towards standardized and verifiable evaluation of multimodal game agents , author =. arXiv preprint arXiv:2604.07429 , year =

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2506.08011 , year =

    Play to generalize: Learning to reason through game play , author =. arXiv preprint arXiv:2506.08011 , year =

    arXiv

  15. [15]

    arXiv preprint arXiv:2505.13426 , year =

    G1: Bootstrapping perception and reasoning abilities of vision-language model via reinforcement learning , author =. arXiv preprint arXiv:2505.13426 , year =

    Pith/arXiv arXiv

  16. [16]

    Advances in Neural Information Processing Systems , volume =

    Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles , author =. Advances in Neural Information Processing Systems , volume =

  17. [17]

    arXiv preprint arXiv:2506.24119 , year =

    Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning , author =. arXiv preprint arXiv:2506.24119 , year =

    arXiv

  18. [18]

    Findings of the Association for Computational Linguistics: EACL 2026 , pages =

    Turn-ppo: Turn-level advantage estimation with ppo for improved multi-turn rl in agentic llms , author =. Findings of the Association for Computational Linguistics: EACL 2026 , pages =

    2026

  19. [19]

    Advances in Neural Information Processing Systems , volume =

    Segment policy optimization: Effective segment-level credit assignment in rl for large language models , author =. Advances in Neural Information Processing Systems , volume =

  20. [20]

    arXiv preprint arXiv:2505.20732 , year =

    Spa-rl: Reinforcing llm agents via stepwise progress attribution , author =. arXiv preprint arXiv:2505.20732 , year =

    Pith/arXiv arXiv

  21. [21]

    Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks , author =

  22. [22]

    Proceedings of the ACM Web Conference 2026 , pages =

    Agentprm: Process reward models for llm agents via step-wise promise and progress , author =. Proceedings of the ACM Web Conference 2026 , pages =

    2026

  23. [23]

    Advances in Neural Information Processing Systems , volume =

    Vagen: Reinforcing world model reasoning for multi-turn vlm agents , author =. Advances in Neural Information Processing Systems , volume =

  24. [24]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Reasoning with language model is planning with world model , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    2023

  25. [25]

    VinePPO: Refining Credit Assignment in RL Training of LLMs , author =

  26. [26]

    arXiv preprint arXiv:2602.05327 , year =

    ProAct: Agentic Lookahead in Interactive Environments , author =. arXiv preprint arXiv:2602.05327 , year =

    arXiv

  27. [27]

    International Conference on Learning Representations , volume =

    On-policy distillation of language models: Learning from self-generated mistakes , author =. International Conference on Learning Representations , volume =

  28. [28]

    International Conference on Learning Representations , volume =

    Minillm: Knowledge distillation of large language models , author =. International Conference on Learning Representations , volume =

  29. [29]

    Thinking Machines Lab: Connectionism , year =

    Kevin Lu and Thinking Machines Lab , title =. Thinking Machines Lab: Connectionism , year =

  30. [30]

    Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages =

    A reduction of imitation learning and structured prediction to no-regret online learning , author =. Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages =. 2011 , organization =

    2011

  31. [31]

    arXiv preprint arXiv:1406.5979 , year =

    Reinforcement and imitation learning via interactive no-regret learning , author =. arXiv preprint arXiv:1406.5979 , year =

    Pith/arXiv arXiv

  32. [32]

    Advances in neural information processing systems , volume =

    Thinking fast and slow with deep learning and tree search , author =. Advances in neural information processing systems , volume =

  33. [33]

    arXiv preprint arXiv:1712.01815 , year =

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm , author =. arXiv preprint arXiv:1712.01815 , year =

    Pith/arXiv arXiv

  34. [34]

    ALFWorld: Aligning Text and Embodied Environments for Interactive Learning , author =

  35. [35]

    Advances in Neural Information Processing Systems , volume =

    Webshop: Towards scalable real-world web interaction with grounded language agents , author =. Advances in Neural Information Processing Systems , volume =

  36. [36]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 technical report , author =. arXiv preprint arXiv:2505.09388 , year =

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2507.06261 , year =

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author =. arXiv preprint arXiv:2507.06261 , year =

    Pith/arXiv arXiv

  38. [38]

    2025 , month = sep, howpublished =

    Introducing Claude Sonnet 4.5 , author =. 2025 , month = sep, howpublished =

    2025

  39. [39]

    2025 , month = nov, howpublished =

    Introducing Claude Opus 4.5 , author =. 2025 , month = nov, howpublished =

    2025

  40. [40]

    2026 , month = feb, howpublished =

    Introducing Claude Sonnet 4.6 , author =. 2026 , month = feb, howpublished =

    2026

  41. [41]

    2026 , month = feb, howpublished =

    Introducing Claude Opus 4.6 , author =. 2026 , month = feb, howpublished =

    2026

  42. [42]

    Proceedings of the AAAI conference on artificial intelligence , volume =

    Deep reinforcement learning with double q-learning , author =. Proceedings of the AAAI conference on artificial intelligence , volume =

  43. [43]

    arXiv preprint arXiv:2601.16725 , year =

    Longcat-flash-thinking-2601 technical report , author =. arXiv preprint arXiv:2601.16725 , year =

    arXiv

  44. [44]

    Intelligence, Physical and Black, Kevin and Brown, Noah and Darpinian, James and Dhabalia, Karan and Driess, Danny and Esmail, Adnan and Equi, Michael and Finn, Chelsea and Fusai, Niccolo and others , journal =

  45. [45]

    nature , volume =

    Human-level control through deep reinforcement learning , author =. nature , volume =. 2015 , publisher =

    2015

  46. [46]

    Artificial Intelligence , volume =

    Sokoban: Enhancing general single-agent search methods using domain knowledge , author =. Artificial Intelligence , volume =. 2001 , publisher =

    2001

  47. [47]

    arXiv preprint arXiv:1312.5602 , year =

    Playing atari with deep reinforcement learning , author =. arXiv preprint arXiv:1312.5602 , year =

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2602.03584 , year =

    V_0 : A Generalist Value Model for Any Policy at State Zero , author =. arXiv preprint arXiv:2602.03584 , year =

    arXiv

  49. [49]

    Learning to Self-Verify Makes Language Models Better Reasoners , author =

  50. [50]

    arXiv preprint arXiv:2501.12948 , year =

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author =. arXiv preprint arXiv:2501.12948 , year =

    Pith/arXiv arXiv

  51. [51]

    International conference on machine learning , pages =

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author =. International conference on machine learning , pages =. 2018 , organization =

    2018

  52. [52]

    Ziebart and Andrew L

    Brian D. Ziebart and Andrew L. Maas and J. Andrew Bagnell and Anind K. Dey , editor =. Maximum Entropy Inverse Reinforcement Learning , booktitle =

  53. [53]

    arXiv preprint arXiv:1707.06347 , year =

    Proximal policy optimization algorithms , author =. arXiv preprint arXiv:1707.06347 , year =

    Pith/arXiv arXiv

  54. [54]

    Advances in neural information processing systems , volume =

    Policy gradient methods for reinforcement learning with function approximation , author =. Advances in neural information processing systems , volume =

  55. [55]

    Advances in neural information processing systems , volume =

    Training language models to follow instructions with human feedback , author =. Advances in neural information processing systems , volume =

  56. [56]

    Proceedings of the nineteenth international conference on machine learning , pages =

    Approximately optimal approximate reinforcement learning , author =. Proceedings of the nineteenth international conference on machine learning , pages =

  57. [57]

    Icml , volume =

    Policy invariance under reward transformations: Theory and application to reward shaping , author =. Icml , volume =. 1999 , organization =

    1999

  58. [58]

    arXiv preprint arXiv:2506.23235 , year =

    Generalist reward models: Found inside large language models , author =. arXiv preprint arXiv:2506.23235 , year =

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2605.16143 , year =

    Look Before You Leap: Autonomous Exploration for LLM Agents , author =. arXiv preprint arXiv:2605.16143 , year =

    Pith/arXiv arXiv