Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Reinforced Language Models for Sequential Decision Making

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that targeted post-training can make a small 3B language model outperform a 72B baseline on sequential decision-making tasks like Frozen Lake.

desk verdict A reasonable GRPO variant for multi-step LLM agents, undermined by a mismatched evaluation protocol that leaves the headline 3B-vs-72B claim unproven. read the letter →

arxiv 2508.10839 v1 pith:QI5UBBB7 submitted 2025-08-14 cs.CL cs.AIcs.LG

Jim Dilkes , Vahid Yazdanpanah , Sebastian Stein This is my paper
classification cs.CLcs.AIcs.LG
keywords multi-stepreinforcementlearningpost-trainingLLMscreditassignmentgroup-relativepolicyoptimizationsequentialdecisionmakinglanguageagentsepisodesamplingFrozenLake
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a 3-billion-parameter language model can be post-trained to outperform a 72-billion-parameter model on a sequential decision-making task, without long reasoning chains. The engine is MS-GRPO, a multi-step extension of GRPO that assigns the full episode reward to every token the model generated during that episode, aided by a sampling scheme that favors episodes with extreme outcomes. If true, targeted post-training becomes a practical alternative to model scale for LLM-based agents. The authors report that the post-trained 3B model beats the 72B baseline by 50% on Frozen Lake, while results on Snake are mixed and the paper's own Discussion attributes the inconsistency to imprecise Monte Carlo credit assignment. A formal Text-Mediated Stochastic Game and Language-Agent Policy framework makes the token-to-action gap explicit and defines which components can be controlled during optimization.

What carries the argument

The load-bearing mechanism is the MS-GRPO objective: episodes are grouped, each episode's composite reward $C_j=\sum_t (r_{j,t}+\Phi_{j,t})$ is normalized within the group to an advantage $A_j$, and the same $A_j$ multiplies every token-level clipped importance ratio $w_{j,t,k}$ in that episode. A second mechanism is Absolute-Advantage-Weighted (AAW) episode sampling, which draws $G' < G$ episodes with probability proportional to $\exp(|A_j|/T_{\mathrm{ep}})$, focusing gradient updates on extreme successes and failures. The Text-Mediated Stochastic Game and Language-Agent Policy formalism supplies the interface: observations are strings, the LLM samples completions, and a parser $\psi$ maps

What would settle it

Run MS-GRPO on a variant of Frozen Lake with reward only on reaching the goal and no intermediate shaping, against a per-step-credit version of the same algorithm; if full-episode credit assignment does not at least match per-step credit on final reward, the stated mechanism is not doing the work. On the paper's own comparison, recompute the 3B versus 72B Frozen Lake gap over all 8 training seeds with the 1,000-episode evaluation protocol; if the post-trained mean does not exceed the 72B baseline mean of 0.38, the scale-substitution claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that MS-GRPO—a multi-step extension of GRPO that assigns the full cumulative episode reward to every token generated in that episode—lets a 3-billion-parameter language model learn sequential decision-making policies from text observations. The authors formalize the setting as a Text-Mediated Stochastic Game and a Language-Agent Policy, separating the environment's text interface from the LLM's token generation and the parser that turns text into actions. On Frozen Lake, the post-trained 3B model reaches a mean reward of $0.57 \pm 0.12$ on the non-slippery training variant, beating the 72B baseline's $0.38 \pm 0.48$ even when that baseline is allowed 4096 tokens;

Load-bearing premise

The load-bearing premise is that giving every generated token the same whole-episode reward, without figuring out which actions actually caused the outcome, still delivers a strong enough learning signal to improve action choice.

Editorial extensions

If this is right

  • Post-trained 3B agent reaches a mean reward of 0.57 on FrozenLake-NotSlippery versus 0.38 for the 72B baseline with a 4096-token budget, showing scale can be traded for targeted training.
  • AAW sampling with G=100 and G'=25 gives roughly 3.5x wall-clock savings over the unsampled version while achieving comparable final reward on Snake.
  • The best Snake-trained agent generalizes to the unseen FrozenLake-Slippery environment with mean reward 0.32, above the DQN baseline's 0.17, suggesting some transfer of learned navigation behavior.
  • A specialized DQN still reaches 4.58 on Snake-Standard versus 0.45 for the best MS-GRPO agent, indicating that for narrow, well-defined tasks a bespoke RL agent remains stronger.
  • Post-trained agents show degraded performance on the poisoned-apple variant, suggesting that reinforcing one skill can override semantic instruction-following, not just improve it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves implicit: on a longer-horizon task with reward only at the end, MS-GRPO's uniform full-episode credit assignment should dilute the signal more than per-step methods, predicting a crossover as episode length grows.
  • The poisoned-apple regression suggests that post-training can overwrite semantic instruction-following; mixing reward-perturbed variants into training could test whether that override is avoidable.
  • Because the LAP framework separates parser, template, and sampler, exploration could be improved by varying generation temperature or templates during training—an avenue the authors mention but do not implement.
  • If the 50% Frozen Lake advantage survives strict 1,000-episode evaluation across all 8 training seeds, it would strengthen the practical case for small post-trained agents over large frozen models in text-observed environments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a formal framework (Text-Mediated Stochastic Game and Language Agent Policy) for LLM-based agents in sequential decision-making environments and proposes MS-GRPO, a GRPO variant that assigns the full cumulative episode reward to every generated token, together with an Absolute-Advantage-Weighted episode sampling strategy. The authors post-train Qwen2.5-3B-Instruct on Snake-Standard and FrozenLake-NotSlippery, compare against Qwen2.5-32B/72B baselines and a DQN, and claim that the post-trained 3B model outperforms the 72B baseline by 50% on Frozen Lake. They also present an ablation of AAW sampling and report mixed generalization results. The Discussion and Conclusions candidly acknowledge high training variance and imprecise Monte Carlo credit assignment as limitations.

Significance. If the headline comparison were statistically valid, the result would be significant: it would show that targeted post-training of a small LLM can substitute for model scale in agentic sequential decision-making, and MS-GRPO plus AAW sampling would be useful practical contributions. The LAP formalization, which separates the LLM, generation configuration, prompt template, and action parser, is a useful conceptual contribution, and the appendix is unusually transparent about hyperparameters and evaluation details. However, the central quantitative claim currently rests on an unmatched evaluation protocol and on selected best runs, so the paper's significance is not yet established.

major comments (3)
  1. [Results/Figure 2; Appendix 'Evaluation Protocol'] The abstract's headline claim ('outperforms a 72B parameter baseline by 50% on the Frozen Lake task') is not supported by the reported comparison. Figure 2 states that MS-GRPO results are averaged over 8 training runs, each evaluated on 50 episodes, while baselines are evaluated on 1,000 episodes. Figure 1's caption confirms that the MS-GRPO evaluations use the same 50 episodes at every step. The appendix fixes the evaluation seed to 0, so the 50-episode set is fixed and is likely a subset of the 1,000-episode baseline distribution. The reported comparison 0.57 ± 0.12 (standard deviation across runs) versus 0.38 ± 0.48 (95% confidence interval over episodes) mixes different estimands and cannot support a 50% improvement. The authors should evaluate all agents on the same episode set with the same number of episodes, or at least report the 72B baseline on the same 50 episodes, and include
  2. [Results, Table 1; Discussion/Conclusions] Several positive claims rely on the single best of 8 training runs. For Snake-Standard the mean final reward is -1.487 with standard deviation 1.093, while the best run achieves 0.45. The statements that the best Snake agent outperforms the 72B model (-1.26 ± 1.80) and that the best Snake-trained agent beats DQN on FrozenLake-Slippery (0.32 vs 0.17 ± 0.38) are selected from a highly variable distribution, with no significance test or multiple-comparison correction. The Discussion/Conclusions themselves attribute the inconsistency to insufficient exploration and an 'imprecise Monte Carlo credit assignment scheme,' which are load-bearing limitations for the method's claimed reliability. The paper should report individual run values (or otherwise characterize the full distribution) and either remove best-run comparisons or explicitly account for selection.
  3. [Appendix 'Hyperparameter Selection'; Figure 3] The AAW sampling ablation is validated on the same metric that was used to choose the hyperparameters. The appendix states that generation temperature, top-k, G, and G' were selected based on evaluation reward on Snake-Standard, and Figure 3 reports AAW ablation results on Snake-Standard reward. This is selection on the test metric: the observed advantage of G=100, G'=25 over other configurations may be a tuning artifact rather than a property of the AAW method. Use a validation split for hyperparameter selection and a separate test split, or demonstrate that the AAW advantage holds on Frozen Lake and the generalization variants.
minor comments (5)
  1. [Introduction / Related Work] The claim of doing something 'for the first time' is too strong given the cited RAGEN work (Wang et al. 2025b), which also assigns credit for entire episodes to the agent's complete action sequence. Please calibrate the novelty claim with respect to RAGEN.
  2. [Methodology, Eq. (1)-(4)] The notation for T_j, |y_j|, and |y_j,t| is used without full definition. In particular, Eq. (1) normalizes by the total number of generated tokens |y_j| while L_CLIP in Eq. (2) sums over tokens at each timestep; make the relationship explicit.
  3. [Experimental Setup / Appendix 'Training Protocol'] The appendix refers to 'FrozenLake-Standard' where the environment is called FrozenLake-NotSlippery. Also, 'each tile has 0.2 probability of being a hole' is imprecise for the standard Gymnasium FrozenLake, which uses a fixed map; clarify whether hole positions are resampled per episode.
  4. [Agent-Environment Interface / Appendix 'Action Parser'] The prompt template allows '1 to 3 actions, separated by the action separator "||"', but the action parser is described as extracting the text inside the first <action>...</action> pair. Explain how multi-action strings are parsed into a sequence of environment actions; otherwise the frequency of invalid actions may be underreported.
  5. [Figure 2 caption] The mixed error-bar convention (standard deviation across runs for MS-GRPO, 95% confidence intervals for baselines) should at least be unified or clearly justified even after the evaluation protocol is matched.

Circularity Check

1 steps flagged · score 3.0 of 10

Mild selection circularity on Snake-Standard and AAW claims; Frozen Lake headline is independent, though its evaluation is asymmetrical.

  1. fitted input called prediction [Technical Appendix, 'Hyperparameter Selection'; used in Results 'AAW Sampling Improves Performance Without Impacting Training Time' and 'Post-training with MS-GRPO Improves Sequential Decision-Making']
    "We determined the selected values for our experiments based on the evaluation reward on Snake-Standard for agents trained on Snake. We found the combination Temperature= 1.5 and k = 3 to give the best mean evaluation reward over 3 runs. Similarly, we used the results from the episode sampling study to determine which values of G and G′ to use, finding that G=100 with G′=25 gave the best combination of training time efficiency and evaluation reward on Snake-Standard."

    The paper fits its generation hyperparameters (temperature, top-k) and its AAW sampling configuration (G, G′) to the Snake-Standard evaluation reward, then reports improvement on that same metric as evidence that MS-GRPO post-training works (Table 1/Figure 1) and that AAW 'improves training performance' (Figure 3). Since the winning configuration was selected on the evaluation metric, the reported Snake-Standard and AAW gains are a consequence of the selection procedure on the test signal, not an independent prediction; the additional comparison of the 'single best-performing run' (Figure 1 caption) against the 72B baseline (0.45 vs −1.26) magnifies the same selection effect. The Frozen Lake claim is less affected because tuning was done on Snake, but the Snake/AAW support claims are parti

full rationale

MS-GRPO is presented as a variation of GRPO with full-episode Monte Carlo credit assignment (Eqs. 1–4). No equation is defined in terms of the experiment's outcome, and the algorithm's behavior does not reduce to its inputs by construction. The central Frozen Lake claim (3B post-trained 0.57±0.12 vs 72B 0.38±0.48, '50%') is empirical rather than definitional: the hyperparameters that produced it were selected on Snake-Standard, not on Frozen Lake, so the headline result retains independent content. However, the supporting Snake and AAW claims contain a real but mild selection circularity: generation hyperparameters (temperature 1.5, top-k 3) and the AAW configuration (G=100, G′=25) were chosen by maximizing evaluation reward on Snake-Standard (appendix 'Hyperparameter Selection'), and the same Snake-Standard evaluation reward is then used as evidence that MS-GRPO improves performance (Table 1, Figure 1) and that AAW sampling improves performance (Figure 3). The additional comparison of the single best of 8 Snake runs against the 72B baseline further selects on the same metric. This makes the Snake/AAW improvements statistically forced rather than independent predictions, though not reductions by construction. There is one self-citation (Li et al. 2025, co-authored by Stein) in the introduction, but it is peripheral and not load-bearing. The headline's evaluation asymmetry (MS-GRPO on 50 fixed-seed episodes vs baselines on 1,000, seed fixed to 0 in the appendix) is a statistical validity concern, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mechanistic entities. TMSG and LAP are formal abstractions, not falsifiable entities. The main pre-theoretical commitments are the reward-shaping design and the assumption that token-level optimization transfers to action-level performance.

free parameters (5)
  • Training generation temperature and top-k = temperature=1.5, top-k=3
    Selected by a sweep over temperature 0.2 to 2.0 and top-k 1 to 10, based on evaluation reward on Snake-Standard (Appendix, Hyperparameter Selection).
  • Episode group sizes G and G' = G=100, G'=25
    Selected from the AAW ablation on Snake-Standard based on evaluation reward and time efficiency (Appendix, Hyperparameter Selection).
  • AAW sampling temperature T_ep = 0.1
    Used to control the softmax weighting over absolute advantages; value is fixed in the training protocol and not swept separately.
  • Reward shaping weights for format penalties = -0.5 invalid action, -0.5 structure, -0.5 extra text, length penalty 0 to -0.5
    Hand-set magnitudes in Reward Design; these shape the composite reward C_j and can dominate the environment reward.
  • KL and clipping coefficients, learning rate = beta=0.1, epsilon_low=epsilon_up=0.1, learning rate=1e-4
    Standard MS-GRPO hyperparameters, reported in the Training Protocol without a dedicated sweep.
assumptions (5)
  • standard math Policy-gradient and clipped importance-sampling theory (Williams 1992; Schulman et al. 2017; Shao et al. 2024) justify the MS-GRPO objective in Equations 1-3.
    The method assumes the standard PPO/GRPO machinery for unbiased or low-variance gradient estimates.
  • domain assumption Optimizing token-level log-probabilities with episode-level advantages improves environment action selection.
    The LAP framework treats the LLM's token distribution as the policy, but there is no proof that token-space gradients align with improvements in the action distribution after parsing.
  • ad hoc to paper The composite reward C_j = sum(r + Phi) with hand-designed format penalties is a suitable training signal that transfers to environment reward.
    Format penalties are introduced for this paper and their weights are arbitrary; the method assumes optimizing C_j also maximizes the environment reward used in evaluation.
  • domain assumption Text observations produced by the TMSG observation function contain all information needed for optimal decisions.
    The agent sees only rendered text plus static rules; if the rendering omits relevant details (e.g., exact enemy snake trajectory), the policy is information-limited.
  • domain assumption Normalizing cumulative rewards across G episodes sharing the same initial state provides a useful baseline for advantage estimation.
    Algorithm 1 samples one initial state per training step and compares all episodes in the group against it; a non-representative initial state can compress or distort advantages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforced Language Models for Sequential Decision Making." pith.science (2026). https://pith.science/paper/QI5UBBB7

@misc{pith2026250810839,
  author       = {Pith},
  title        = {Pith review of: Reinforced Language Models for Sequential Decision Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QI5UBBB7}},
  note         = {Machine review of arXiv:2508.10839}
}
read the original abstract

Large Language Models (LLMs) show potential as sequential decision-making agents, but their application is often limited due to a reliance on large, computationally expensive models. This creates a need to improve smaller models, yet existing post-training methods are designed for single-turn interactions and cannot handle credit assignment in multi-step agentic tasks. To address this, we introduce Multi-Step Group-Relative Policy Optimization (MS-GRPO), a new algorithm for post-training LLM agents, grounded in formal Text-Mediated Stochastic Game (TSMG) and Language-Agent Policy (LAP) frameworks. For credit assignment, MS-GRPO attributes the entire cumulative episode reward to each individual episode step. We supplement this algorithm with a novel absolute-advantage-weighted episode sampling strategy that we show improves training performance. We evaluate our approach by post-training a 3-billion parameter model on Snake and Frozen Lake. Our experiments demonstrate that the method is effective in improving decision-making performance: our post-trained 3B parameter model outperforms a 72B parameter baseline by 50% on the Frozen Lake task. This work demonstrates that targeted post-training is a practical and efficient alternative to relying on model scale for creating sequential decision-making agents using LLMs.

Figures

Figures reproduced from arXiv: 2508.10839 by the authors.

Figure 1
Figure 1. Training progression over 700 steps across four [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Mean reward per episode for MS-GRPO post [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Ablation study on convergence and training ef [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    The paper organizes research on generalist game AI into Dataset, Model, Harness, and Benchmark pillars and charts a five-level progression from single-game mastery to agents that create and live inside game multiverses.

  2. Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse

    cs.CV 2026-05 unverdicted novelty 3.0 of 10

    This work traces four eras of generalist game players across dataset, model, harness, and benchmark pillars and charts a five-level roadmap ending in agents that create and evolve within game multiverses.

Reference graph

Works this paper leans on

43 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

    work page

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bani-Harouni, D.; Pellegrini, C.; Özsoy, E.; Keicher, M.; and Navab, N. 2025. Language Agents for Hypothesis -driven Clinical Decision Making with Reinforcement Learning . ArXiv:2506.13474 [cs]

    arXiv 2025

  4. [4]

    Bernard, D.; Dorais, G.; Fry, C.; Gamble, E.; Kanefsky, B.; Kurien, J.; Millar, W.; et al. 1998. Design of the Remote Agent experiment for spacecraft autonomy. In 1998 IEEE Aerospace Conference Proceedings ( Cat . No . 98TH8339 ) , volume 2, 259--281 vol.2. ISSN: 1095-323X

    work page 1998

  5. [5]

    DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; et al. 2025. DeepSeek - R1 : Incentivizing Reasoning Capability in LLMs via Reinforcement Learning . ArXiv:2501.12948 [cs]

    arXiv 2025

  6. [6]

    Evans, R.; and Gao, J. 2016. DeepMind AI Reduces Google Data Centre Cooling Bill by 40\ Accessed: 2025-08-02. https://deepmind.google/discover/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-by-40/

    work page 2016

  7. [7]

    Georgeff, M. 1988. A Theory of Action for MultiAgent Planning . In Bond, A. H.; and Gasser, L., eds., Readings in Distributed Artificial Intelligence , 205--209. Morgan Kaufmann. ISBN 978-0-934613-63-7

    work page 1988

  8. [8]

    A.; Bernstein, D

    Hansen, E. A.; Bernstein, D. S.; and Zilberstein, S. 2004. Dynamic programming for partially observable stochastic games. In Proceedings of the 19th national conference on Artifical intelligence , AAAI '04, 709--715. San Jose, California: AAAI Press. ISBN 978-0-262-51183-4

    work page 2004

Show all 43 references
  1. [9]

    Hou, Z.; Lv, X.; Lu, R.; Zhang, J.; Li, Y.; Yao, Z.; Li, J.; et al. 2025. T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling . ArXiv:2501.11651 [cs]

    2025 arXiv

  2. [10]

    Kambhampati, S.; Valmeekam, K.; Guan, L.; Verma, M.; Stechly, K.; Bhambri, S.; Saldyt, L.; and Murthy, A. 2024. Position: LLMs can't plan, but can help planning in LLM -modulo frameworks. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of I...

    2024

  3. [11]

    Kamradt, G. 2025. Snake Bench : Competitive Snake Game Simulation with LLMs . Accessed: 2025-08-02. https://github.com/gkamradt/SnakeBench

    2025

  4. [12]

    Kool, W.; Van Hoof, H.; and Welling, M. 2019. Attention, learn to solve routing problems! In 7th International Conference on Learning Representations , ICLR 2019, May 6, 2019 - May 9, 2019 , 7th International Conference on Learning Representations , ICLR 2019. New Orleans, LA,...

    2019

  5. [13]

    Levine, S.; Finn, C.; Darrell, T.; and Abbeel, P. 2016. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(Compendex). Publisher: Microtome Publishing

    2016

  6. [14]

    Li, Z.; Wu, W.; Wang, Y.; Xu, Y.; Hunt, W.; and Stein, S. 2025. HMCF : A Human -in-the-loop Multi - Robot Collaboration Framework Based on Large Language Models . ArXiv:2505.00820 [cs]

    2025 arXiv

  7. [15]

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Playing Atari with Deep Reinforcement Learning . ArXiv:1312.5602 [cs]

    2013 arXiv

  8. [16]

    E.; Wan, Y.; Yu, H.; Grosnit, A.; Gonzalez-Billandon, J.; Zimmer, M.; Wang, J.; et al

    Mower, C. E.; Wan, Y.; Yu, H.; Grosnit, A.; Gonzalez-Billandon, J.; Zimmer, M.; Wang, J.; et al. 2024. ROS - LLM : A ROS framework for embodied AI with task feedback and structured reasoning. ArXiv:2406.19741 [cs]

    2024 arXiv

  9. [17]

    Murphy, S. A. 2003. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(2): 331--355

    2003

  10. [18]

    P.; Perelman, A.; Ramesh, A.; Clark, A.; et al

    OpenAI; Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; et al. 2024. GPT -4o System Card . ArXiv:2410.21276 [cs]

    2024 arXiv

  11. [19]

    L.; Mishkin, P.; Zhang, C.; et al

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS '22, 277...

    2022

  12. [20]

    E.; Zhang, K.; and Kim, J.-K

    Park, C.; Han, S.; Guo, X.; Ozdaglar, A. E.; Zhang, K.; and Kim, J.-K. 2025. MAPoRL : Multi - Agent Post - Co - Training for Collaborative Large Language Models with Reinforcement Learning . In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., Proceedings of the 63...

    2025

  13. [21]

    Qwen; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; et al. 2025. Qwen2.5 Technical Report . ArXiv:2412.15115 [cs]

    2025 arXiv

  14. [22]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2023. Direct Preference Optimization : Your Language Model is Secretly a Reward Model . Advances in Neural Information Processing Systems, 36: 53728--53741

    2023

  15. [23]

    Schaul, T.; Quan, J.; Antonoglou, I.; and Silver, D. 2016. Prioritized experience replay. In 4th International Conference on Learning Representations , ICLR 2016, May 2, 2016 - May 4, 2016 , 4th International Conference on Learning Representations , ICLR 2016 - Conference Trac...

    2016

  16. [24]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms . ArXiv:1707.06347 [cs]

    2017 arXiv

  17. [25]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; et al. 2024. DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models . ArXiv:2402.03300 [cs]

    2024 arXiv

  18. [26]

    Shapley, L. S. 1953. Stochastic Games *. Proceedings of the National Academy of Sciences, 39(10): 1095--1100. Publisher: Proceedings of the National Academy of Sciences

    1953

  19. [27]

    Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 8634--8652

    2023

  20. [28]

    S.; and Barto, A

    Sutton, R. S.; and Barto, A. G. 2018. Reinforcement Learning : An Introduction . Cambridge, MA, USA: A Bradford Book. ISBN 978-0-262-03924-6

    2018

  21. [29]

    Tanahashi, K.; Inoue, Y.; Yamaguchi, Y.; Yaginuma, H.; Shiotsuka, D.; Shimatani, H.; Iwamasa, K.; et al. 2023. Evaluation of Large Language Models for Decision Making in Autonomous Driving . ArXiv:2312.06351 [cs]

    2023 arXiv

  22. [30]

    U.; Cola, G

    Towers, M.; Kwiatkowski, A.; Terry, J.; Balis, J. U.; Cola, G. D.; Deleu, T.; Goulão, M.; et al. 2024. Gymnasium: A Standard Interface for Reinforcement Learning Environments . ArXiv:2407.17032 [cs]

    2024 arXiv

  23. [31]

    Traum, D. R. 1999. Speech Acts for Dialogue Agents . In Wooldridge, M.; and Rao, A., eds., Foundations of Rational Agency , 169--201. Dordrecht: Springer Netherlands. ISBN 978-94-015-9204-8

    1999

  24. [32]

    Trivedi, H.; Khot, T.; Hartmann, M.; Manku, R.; Dong, V.; Li, E.; Gupta, S.; et al. 2024. AppWorld : A Controllable World of Apps and People for Benchmarking Interactive Coding Agents . In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting o...

    2024

  25. [33]

    Wang, Y.; Yang, Q.; Zeng, Z.; Ren, L.; Liu, L.; Peng, B.; Cheng, H.; et al. 2025 a . Reinforcement Learning for Reasoning in Large Language Models with One Training Example . ArXiv:2504.20571 [cs]

    2025 arXiv

  26. [34]

    Wang, Z.; Wang, K.; Wang, Q.; Zhang, P.; Li, L.; Yang, Z.; Yu, K.; et al. 2025 b . RAGEN : Understanding Self - Evolution in LLM Agents via Multi - Turn Reinforcement Learning . ArXiv:2504.20073 [cs]

    2025 arXiv

  27. [35]

    Williams, R. J. 1992. Simple Statistical Gradient - Following Algorithms for Connectionist Reinforcement Learning . Mach. Learn., 8(3-4): 229--256

    1992

  28. [36]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct : Synergizing Reasoning and Acting in Language Models . ArXiv:2210.03629 [cs]

    2023 arXiv

  29. [37]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; et al. 2025. DAPO : An Open - Source LLM Reinforcement Learning System at Scale . ArXiv:2503.14476 [cs]

    2025 arXiv

  30. [38]

    B.; Shu, T.; and Gan, C

    Zhang, H.; Du, W.; Shan, J.; Zhou, Q.; Du, Y.; Tenenbaum, J. B.; Shu, T.; and Gan, C. 2024. Building Cooperative Embodied Agents Modularly with Large Language Models . ArXiv:2307.02485 [cs]

    2024 arXiv

  31. [39]

    Zheng, C.; Liu, S.; Li, M.; Chen, X.-H.; Yu, B.; Gao, C.; Dang, K.; et al. 2025 a . Group Sequence Policy Optimization . ArXiv:2507.18071 [cs]

    2025 arXiv

  32. [40]

    Zheng, Y.; Fu, D.; Hu, X.; Cai, X.; Ye, L.; Lu, P.; and Liu, P. 2025 b . DeepResearcher : Scaling Deep Research via Reinforcement Learning in Real -world Environments . ArXiv:2504.03160 [cs]

    2025 arXiv

  33. [41]

    Zhong, H.; Shan, Z.; Feng, G.; Xiong, W.; Cheng, X.; Zhao, L.; He, D.; et al. 2025. DPO Meets PPO : Reinforced Token Optimization for RLHF . ArXiv:2404.18922 [cs]

    2025 arXiv

  34. [42]

    Zhou, R.; Du, S.; and Li, B. 2024. Reflect- RL : Two - Player Online RL Fine - Tuning for LMs . In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , 995--1015. Ban...

    2024

  35. [43]

    M.; Stiennon, N.; Wu, J.; Brown, T

    Ziegler, D. M.; Stiennon, N.; Wu, J.; Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P.; and Irving, G. 2020. Fine- Tuning Language Models from Human Preferences . ArXiv:1909.08593 [cs]

    2020 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.