Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Experience sharing across a swarm of small language models raises cumulative RL post-training reward by up to 94% over isolated training.

desk verdict The idea is promising but the headline 94% gain is confounded by zero-advantage filtering, and the experiment needs an equal-footing baseline and multiple seeds. read the letter →

arxiv 2509.08721 v1 pith:WIXV5TB2 submitted 2025-09-10 cs.LG cs.MA

Jeffrey Amico , Gabriel Passamani Andrade , John Donaghy , Ben Fielding , Tristin Forbus , Harry Grieve , Semih Kara , Jari Kolehmainen
show 7 more authors
Yihua Lou Christopher Nies Edward Phillip Flores Nuño Diogo Ortega Shikhar Rastogi Austin Virts Matthew J. Wright
This is my paper · ORCID
classification cs.LGcs.MA
keywords decentralizedreinforcementlearningexperiencesharinglanguagemodelpost-trainingswarmintelligenceGRPOverifiablerewardssmallmodelsrolloutsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SAPO (Swarm sAmpling Policy Optimization), a fully decentralized and asynchronous reinforcement-learning post-training algorithm in which each node trains its own policy but can build its training batch by mixing its own rollouts with rollouts shared by other nodes. The central claim is that collective experience sharing lets useful discoveries—what the authors call 'aha moments'—propagate across the swarm, improving sample efficiency and final task reward. In controlled experiments with eight small language models trained on procedurally generated reasoning tasks over 2000 rounds, a balanced configuration mixing four local and four external rollouts per node accumulated 1093.31 cumulative reward versus 561.79 for a no-sharing baseline, a 94% improvement. The paper also reports that the benefit is not unlimited: relying too heavily on external rollouts produces oscillation and forgetting, and a large open demo indicates the benefit is model-dependent. If correct, the work offers a route to RL post-training on heterogeneous, loosely connected hardware without weight synchronization.

What carries the argument

The operative mechanism is decoded-text rollout sharing with local training-batch assembly. Each node $n$ keeps its own policy $\pi_n$ and, each round, samples questions, generates eight completions per question, and broadcasts a subset of them as (question, ground truth, rollouts, metadata) tuples. The node then builds its training set from $I_n$ self-rollouts and $J_n$ external rollouts drawn from the shared pool—discarding zero-advantage samples in these experiments—and performs a policy-gradient update (GRPO, no KL penalty, asymmetric clipping). Setting $J_n = 0$ recovers standard single-agent RL fine-tuning, making the local/external ratio the knob under study.

What would settle it

Run the same eight-agent 2000-round experiment but allow the no-sharing baseline to discard zero-advantage rollouts from its own batch before the GRPO update, keeping the total number of training samples fixed; if its cumulative reward rises to roughly 1093.31, the measured benefit is attributable to sample filtering rather than to cross-agent experience sharing.

Watch

Extended reading notes

Core claim

The paper's central discovery is that in RL post-training for reasoning, sharing decoded rollouts among independently updated policies can substantially accelerate learning. Formally, each node broadcasts tuples of a question, its verifiable ground-truth answer, its rollout completions, and verification metadata; nodes then subsample $I_n$ local and $J_n$ external datapoints for GRPO updates, keeping total samples per round fixed. The empirical finding is a non-monotone effect: the 4-local/4-external mix yields 1093.31 cumulative reward, 94% above the 561.79 no-sharing baseline, while 2-local/6-external gives 945.87 with pronounced oscillations and 6-local/2-external gives 854.43. The authors attribute the gain to propagation of rare successful solution patterns and note that the flexibility to discard zero-advantage rollouts is built into SAPO, a flexibility the baseline lacks.

Load-bearing premise

The 94% gain could come entirely from the freedom to discard zero-advantage rollouts, because the no-sharing baseline is forced to keep all of its own samples while SAPO configurations filter a larger mixed pool.

Editorial extensions

If this is right

  • RL post-training can run over decentralized, heterogeneous, loosely connected compute without weight synchronization, lowering the infrastructure barrier for community and edge training.
  • Rare successful reasoning patterns ('aha moments') propagate across agents quickly, so formats and solution conventions spread without explicit reward shaping; the authors dropped their formatting reward after observing this.
  • There is a nontrivial optimal ratio of local to external rollouts, so practitioners should treat the mix as a hyperparameter, since too much external reliance degrades training stability.
  • The benefit of sharing appears strongest for mid-capacity models that absorb and re-emit diverse rollouts, while stronger models in the demo did not improve; sampling strategy and model capacity interact.
  • The paper predicts that with better-than-uniform sampling from the shared pool, even stronger models could benefit from swarm participation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If zero-advantage filtering is the true driver, SAPO's contribution is better described as sample curation, and a single agent discarding its own low-reward rollouts could reproduce the gain; a filtered no-sharing control would settle this.
  • The oscillation in the 2-local/6-external configuration resembles coupled-oscillator dynamics, suggesting an adaptive scheme that tunes the external fraction during training as a testable stability fix.
  • Because rollouts are shared as plain text, the framework extends to heterogeneous modalities and even non-trained generators, offering an experimental route to study social learning in model populations.
  • The demo's model-dependent result hints that a policy's capacity gates the value of external experience; ranking agents by an 'absorptive capacity' measure could enable targeted sharing and fairer aggregation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous RL post-training algorithm in which each node trains its own policy while sharing decoded rollouts (plain-text question-answer samples) with other nodes. Each node assembles its training set by combining local rollouts with external rollouts sampled from the swarm, then performs a standard policy-gradient update (GRPO in the experiments). The authors report controlled experiments with eight Qwen2.5-0.5B agents trained on ReasoningGYM tasks for 2000 rounds, comparing no-sharing (8 local / 0 external) against three sharing configurations (6/2, 4/4, 2/6). They report that the 4/4 configuration achieves a cumulative reward of 1093.31 versus 561.79 for the baseline, a 94% improvement. They also present observations from a large-scale community demo involving thousands of nodes on heterogeneous hardware, where swarm-trained Qwen2.5-0.5B models outperform an isolated counterpart after roughly 175 normalized rounds, while stronger models show no clear benefit.

Significance. The core idea—sharing decoded rollouts across independently trained policies to bootstrap reasoning—is simple, plausible, and practically appealing. The decentralized, asynchronous formulation avoids the synchronization and communication bottlenecks of centralized distributed RL and naturally accommodates heterogeneous nodes. The paper is clearly written and the algorithm is straightforward to describe. If the empirical effect were established, SAPO would be a useful contribution to multi-agent RL post-training. However, the current evidence does not yet support the headline quantitative claim: the controlled comparison confounds experience sharing with zero-advantage filtering, the uncertainty quantification is inadequate (single run, min/max across agents as 'confidence intervals'), and the reported 94% is selected as the best of four configurations. The large-scale demo is uncontrolled and better treated as anecdotal. The conceptual contribution deserves further validation, but the experimental design must be strengthened before the central claim can be accepted.

major comments (3)
  1. [Section 5] The central comparison is confounded: the SAPO configurations filter zero-advantage rollouts before sampling external experience, while the baseline is explicitly not allowed this operation ('Note that SAPO gives agents the flexibility to subsample from a larger pool and remove the uninformative, 0-advantage samples, which the baseline cannot do.'). Consequently the headline comparison (1093.31 vs 561.79) differs in two variables: external rollouts and zero-advantage filtering. The authors' own Section 6 observation that unfiltered uniform sampling over-represents zero-advantage rollouts and degrades performance suggests filtering may be the active ingredient. A controlled baseline that is allowed the same zero-advantage filtering, and/or a SAPO variant that does not filter, is required to isolate the contribution of experience sharing.
  2. [Section 5, Figure 2] The paper reports no repeated seeds, and the 'confidence intervals' in Figure 2 are described as 'the minimum and maximum across agents,' which is a spread across agents in a single run, not an uncertainty estimate for the training procedure. The headline 94% improvement is the cumulative reward of one run of the best of four configurations, so the magnitude is also subject to selection over configurations. Multiple independent runs with proper standard errors (or full per-seed trajectory plots) are needed before a quantitative claim of 94% can be supported.
  3. [Section 5] The term 'rollout' is used ambiguously. The baseline '8 local / 0 external rollouts' could mean 8 question-level sets of 8 completions each, while 'samples J rollouts from the remaining ones' in the SAPO description suggests J individual completions. This ambiguity directly affects whether the total number of training samples is actually fixed across configurations, as claimed ('ensuring the total number of training samples was fixed across all setups'). The authors should define 'rollout' precisely (a (question, set-of-completions) object or a single completion) and report the per-agent update batch size in each configuration; otherwise the comparability of cumulative rewards is not established.
minor comments (6)
  1. [Section 3.2 and Algorithm 1] Algorithm 1 calls SampleSelf and SampleExternal without specifying the zero-advantage filtering procedure that is described only in Section 5. The algorithm description should match the experimental procedure, either by including the filter in the pseudocode or by stating that filtering is a configurable sampling strategy.
  2. [Section 5, Figure 2] The label 'confidence intervals' in the Figure 2 caption is misleading; the shaded region is the min/max range across agents. Relabel it as 'range across agents' or use a proper statistical uncertainty estimate.
  3. [Section 6] The sentence 'statistical testing confirms that after roughly 175 normalized rounds, the swarm-trained variant outperforms its isolated counterpart' does not specify the test, the multiple-testing correction, or the data pre-processing. Provide details of the statistical procedure and the definition of 'adjusted p-value'.
  4. [Throughout] There are several typos and wording issues: 'adversely effected' should be 'adversely affected' (Section 5); 'Complimentary' should be 'Complementary' (Section 7); 'heterogenous' should be 'heterogeneous' (appears multiple times).
  5. [Section 4.3] The claim that experience sharing made a formatting reward unnecessary is anecdotal. If retained, it should be supported by a comparison or removed.
  6. [Section 6] The large-scale demo is uncontrolled: nodes voluntarily participate, differ in hardware and model configurations, and the comparison between swarm and isolated training may be affected by self-selection. The authors should frame these results as observational insights rather than evidence for the algorithm, which they mostly do, but the current wording ('Our findings show that swarm-based training can yield measurable gains') could be read too strongly.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity; the 94% claim is an empirical comparison, though Section 5's zero-advantage filtering asymmetry is a validity confound rather than a circular step.

full rationale

SAPO is defined operationally (Algorithm 1) as decentralized rollout sharing plus optional subsampling; no quantity in the paper is fitted from the evaluation metric and then renamed as a prediction, and no uniqueness or optimality theorem is imported from the authors' prior work to force the algorithm. The headline result (Section 5: 1093.31 vs 561.79, '94% improvement') is a measured cumulative reward over ReasoningGYM training, not a consequence of the definitions; it is therefore not circular. The explicit statement that SAPO 'can remove the uninformative, 0-advantage samples, which the baseline cannot do' does identify a genuine experimental confound: the comparison varies two mechanisms at once (external sampling and zero-advantage filtering), and in GRPO filtering can alter group advantage normalization, so the causal attribution to 'experience sharing' is not isolated. That is a correctness/validity risk, not a self-referential derivation. The Gensyn/GenRL self-citations describe the authors' platform used in the demo and are not load-bearing evidence for the controlled results. Because the central empirical claim has independent content and no prediction reduces by construction, the circularity score is low despite the experimental-design concerns.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the choice of the 4/4 local/external split, the zero-advantage filtering rule, and the validity of off-policy re-encoding. The paper introduces no new physical entities; the 'swarm' is a network of standard policies. The key unstated assumptions are that off-policy re-encoding is valid and that the 8-agent setup reflects larger swarms.

free parameters (5)
  • Local/external rollout split (I_n, J_n) = Best configuration 4/4 (4 local, 4 external)
    The paper evaluates four configurations and reports the best one (4/4) as the headline 94% gain. The choice of 4/4 is selected post hoc from the data.
  • Number of completions per question L_n = 8
    Fixed across all experiments; affects the effective batch size and the variance of the advantage estimates.
  • Training rounds = 2000
    The cumulative reward is measured over 2000 rounds; early stopping or longer training could change the comparison.
  • GRPO hyperparameters = lr=0.001, epsilon_low=0.2, epsilon_high=0.28, KL weight=0
    Standard GRPO hyperparameters, selected in initial experiments (KL weight set to zero following DAPO).
  • Zero-advantage filtering = discard all rollouts with advantage 0
    Only SAPO configurations use this filter; the baseline cannot. This confounds the sharing benefit with the filtering benefit.
assumptions (4)
  • domain assumption GRPO policy gradient improves the policy when rewards are accurate
    The paper relies on GRPO as the underlying RL update without proving convergence or boundedness in the shared-data setting.
  • domain assumption Re-encoding text rollouts from other policies yields valid token-level values for the local policy
    The method requires that off-policy rollouts can be re-scored under the current policy; this is a standard importance-weighting assumption but unstated in detail.
  • domain assumption ReasoningGYM verifiers provide correct reward signals
    All rewards come from programmatic verifiers; errors in parsing or partial credit would bias the training signal.
  • domain assumption The 8-agent controlled experiment is representative of larger swarms
    The headline 94% is from 8 homogeneous agents with synchronized rounds; scaling to thousands of heterogeneous nodes is only shown in an uncontrolled demo.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing." pith.science (2026). https://pith.science/paper/WIXV5TB2

@misc{pith2026250908721,
  author       = {Pith},
  title        = {Pith review of: Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIXV5TB2}},
  note         = {Machine review of arXiv:2509.08721}
}
read the original abstract

Post-training language models (LMs) with reinforcement learning (RL) can enhance their complex reasoning capabilities without supervised fine-tuning, as demonstrated by DeepSeek-R1-Zero. However, effectively utilizing RL for LMs requires significant parallelization to scale-up inference, which introduces non-trivial technical challenges (e.g. latency, memory, and reliability) alongside ever-growing financial costs. We present Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous RL post-training algorithm. SAPO is designed for decentralized networks of heterogenous compute nodes, where each node manages its own policy model(s) while "sharing" rollouts with others in the network; no explicit assumptions about latency, model homogeneity, or hardware are required and nodes can operate in silo if desired. As a result, the algorithm avoids common bottlenecks in scaling RL post-training while also allowing (and even encouraging) new possibilities. By sampling rollouts "shared" across the network, it enables "Aha moments" to propagate, thereby bootstrapping the learning process. In this paper we show SAPO achieved cumulative reward gains of up to 94% in controlled experiments. We also share insights from tests on a network with thousands of nodes contributed by Gensyn community members running the algorithm on diverse hardware and models during an open-source demo.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 9 canonical work pages

  1. [1]

    Self-play fine-tuning converts weak language models to strong language models, 2024

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL https://arxiv.org/abs/2401.01335

    arXiv 2024

  2. [2]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  3. [3]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. URL https://arxiv.org/abs/2305.14325

    arXiv 2023

  4. [4]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL https://arxiv.org/abs/2505.24298

    arXiv 2025

  5. [5]

    On designing effective rl reward at training time for llm reasoning, 2024

    Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for llm reasoning, 2024. URL https://arxiv.org/abs/2410.15115

    arXiv 2024

  6. [6]

    Introducing rl swarm’s new backend: Genrl

    Gensyn . Introducing rl swarm’s new backend: Genrl. https://www.gensyn.ai/articles/genrl, 2025. Accessed: 2025-08-28

    work page 2025

  7. [7]

    Gensyn rl swarm

    Gensyn. Gensyn rl swarm. https://github.com/gensyn-ai/rl-swarm, 2025. Accessed: 2025-08-20

    work page 2025

  8. [8]

    Clipscore: A reference-free evaluation metric for image captioning, 2022

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022. URL https://arxiv.org/abs/2104.08718

    arXiv 2022

Show all 41 references
  1. [9]

    Bowman, Tim Rockt\" a schel, and Ethan Perez

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rockt\" a schel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers. In Proceedings of the 41st International Conf...

    2024

  2. [10]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2025 arXiv

  3. [11]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural ...

    2022

  4. [12]

    Camel: Communicative agents for "mind" exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Pro...

    2023

  5. [13]

    Improving multi-agent debate with sparse communication topology, 2024

    Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. Improving multi-agent debate with sparse communication topology, 2024. URL https://arxiv.org/abs/2406.11776

    2024 arXiv

  6. [14]

    Encouraging divergent thinking in large language models through multi-agent debate, 2024

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate, 2024. URL https://arxiv.org/abs/2305.19118

    2024 arXiv

  7. [15]

    Marft: Multi-agent reinforcement fine-tuning, 2025

    Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. Marft: Multi-agent reinforcement fine-tuning, 2025. URL https://arxiv.org/abs/2504.16129

    2025 arXiv

  8. [16]

    Llm collaboration with multi-agent reinforcement learning, 2025

    Shuo Liu, Zeyu Liang, Xueguang Lyu, and Christopher Amato. Llm collaboration with multi-agent reinforcement learning, 2025. URL https://arxiv.org/abs/2508.04652

    2025

  9. [17]

    Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning

    Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhan...

    2024

  10. [18]

    Mistral-AI, :, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Saga...

    2025 arXiv

  11. [19]

    Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent llm training, 2025. URL https://arxiv.org/abs/2412.01928

    2025

  12. [20]

    Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul Kim, Hanieh Deilamsalehy, Namyong Park, Mike Rimer, Zhehao Zhang, Huanrui Yang, Ryan A

    Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zhengmian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, Junda Wu, Ashish Singh, Yu Wang, Jiuxiang Gu, Franck Dernoncourt, Nesreen K. Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul K...

    2024 arXiv

  13. [21]

    Aligning language models to follow instructions

    OpenAI. Aligning language models to follow instructions. https://openai.com/index/instruction-following/, 2022. Accessed: 2025-08-19

    2022

  14. [22]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, 2024. Accessed: 2025-08-19

    2024

  15. [23]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  16. [24]

    Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning, 2025

    Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning, 2025. URL https://arxiv.org/abs/2502.18439

    2025 arXiv

  17. [25]

    Red teaming language models with language models, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022. URL https://arxiv.org/abs/2202.03286

    2022 arXiv

  18. [26]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team . Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  19. [27]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  20. [28]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  21. [29]

    Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025

    Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025. URL https://arxiv.org/abs/2505.24760

    2025

  22. [30]

    Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch

    Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains, 2025. URL https://arxiv.org/abs/2501.05707

    2025 arXiv

  23. [31]

    Fine-tuning language models for factuality

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher Manning, and Chelsea Finn. Fine-tuning language models for factuality. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https://openreview.net/forum?id=kEK08VdSO5

    2023

  24. [32]

    Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale llm training, 2025

    Bo Wu, Sid Wang, Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, Xiaocheng Tang, Yundi Qian, Beibei Zhu, and Rui Hou. Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale llm traini...

    2025 arXiv

  25. [33]

    Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023. URL https...

    2023 arXiv

  26. [34]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  27. [35]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, X...

    2025 arXiv

  28. [36]

    Generative verifiers: Reward modeling as next-token prediction, 2025

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction, 2025. URL https://arxiv.org/abs/2408.15240

    2025 arXiv

  29. [37]

    Sirius: Self-improving multi-agent systems via bootstrapped reasoning, 2025

    Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, and James Zou. Sirius: Self-improving multi-agent systems via bootstrapped reasoning, 2025. URL https://arxiv.org/abs/2502.04780

    2025 arXiv

  30. [38]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593

    2020 arXiv

  31. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  32. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  33. [41]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.