Pith. sign in

REVIEW 4 major objections 5 minor 75 references

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that using a single frozen LLM as a user simulator in multi-agent RL causes the policy to overfit to the simulator's dominant response mode, collapse its own entropy, and transfer poorly to unseen simulators and real…

desk verdict A strong, referee-worthy paper whose practical claim holds up, but the mode-collapse mechanism is asserted more strongly than the evidence supports. read the letter →

arxiv 2608.12253 v1 pith:J7VUG42J submitted 2026-08-12 cs.CL cs.AIcs.LG

Simon Yu , Nicholas Tomlin , Marwa Abdulhai , Ximing Lu , Derek Chong , Abe Hou , Dilara Soylu , Sergey Levine
show 2 more authors
Christopher D. Manning Weiyan Shi
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords simulatorcollapsemulti-agentreinforcementlearningLLMusersimulationmodepolicyentropyco-trainingverbalizedsamplingpopulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning of an LLM agent against a single frozen LLM user simulator systematically fails to generalize, and this paper claims the failure is structural rather than an optimizer bug. The mechanism is simulator collapse: aligned LLMs concentrate probability on a modal response, so the policy gradient is dominated by that one behavior, the agent learns the narrow strategy that beats the simulator's mode, its entropy collapses, and the policy transfers poorly to unseen simulators and real users. The paper formalizes the collapse as a gradient-bias bound and an entropy-concentration result, then gives two complementary fixes: Verbalized Sampling, which widens the simulator's per-turn distribution at inference time, and Co-Training, which trains the simulator together with the policy so the mode keeps moving. On Persuasion for Good, $\tau^2$-bench, and CooperBench, both fixes recover most of the held-out gap, Population Co-Training does best, and a human study confirms real-user gains. If right, the conclusion is that environment diversity, not only policy diversity, is a requirement for multi-turn RL to transfer to deployment.

What carries the argument

The load-bearing object is the per-turn collapse error $\epsilon_\phi(s,a^\pi)$, the probability that the simulator $\phi$ deviates from its most likely response at a visited history, accumulated over the horizon as $\bar\epsilon_H(\theta)=\mathbb{E}[\sum_{t=1}^H \epsilon_\phi(s_t,a^\pi_t)]$. Theorem 3.2 couples the real simulator rollout to a deterministic mode-user rollout and bounds the resulting gradient bias by $2BR_{\max}\bar\epsilon_H(\theta)$; Lemma 3.3 shows collapse removes simulator-side reward variance from the group-relative advantage; Proposition 3.4 and Corollary 3.5 convert that persistent mode advantage into geometric concentration of strategy mass on the mode-exploit set $A_x$; Proposition 3.6 converts the concentration into a lower bound on deployment regret when real-user behaviors fall outside $A_x$. The two proposed fixes attack the chain at different points: Verbalized Sampling replaces the collapsed token distribution with a verbalized distribution that approximates the simulator's pre-aligned reference distribution, and Co-Training continually re-randomizes the exploit set so the geometric concentration has no fixed target.

What would settle it

Estimate $\epsilon_\phi(s_t,a^\pi_t)$ directly on states reached by the training policy: for a sample of visited histories, draw multiple simulator responses and compute the average probability of deviating from the modal response. If that average is large (roughly $>1/H$, making $\bar\epsilon_H(\theta)$ non-negligible), Theorem 3.2's bound is vacuous and the predicted collapse should not occur; conversely, if a deliberately de-modalized simulator that is forced to respond diversely at visited states still produces the peak-then-degrade OOD curve, the collapse story would need revision. The paper's own proxy, zero-variance batch fraction, cannot settle this because sparse rewards and all-failure batches also give zero variance.

Watch

Extended reading notes

Core claim

At the paper's core is a negative discovery with a mechanism attached: an LLM policy trained against a single frozen LLM user simulator does not merely underperform, it collapses. The paper defines the simulator's mode at each turn as its most likely response, and defines simulator collapse as a small probability of deviating from that mode on the histories the policy actually visits. It then proves that under such collapse the REINFORCE gradient differs from the gradient against a deterministic mode-emitting simulator by at most $2BR_{\max}\bar\epsilon_H(\theta)$, where $\bar\epsilon_H(\theta)$ is the accumulated per-turn collapse error; simulator-side reward variance disappears, so group-relative advantages rank trajectories by how well they exploit the mode; a KL-regularized softmax update then concentrates policy mass on the mode-exploit set geometrically fast, producing a low-entropy policy with bounded deployment regret against users whose behaviors lie outside that set. The fixes follow from the mechanism: Verbalized Sampling makes the simulator sample from a verbalized response distribution, approximately recovering the pre-alignment reference gradient, while Co-Training makes the simulator a moving target by updating it on the same rollouts so no fixed exploit set accumulates odds. The empirical section shows the predicted peak-then-collapse training curves, recovery on the held-out simulator panel, and transfer of the gains to human users.

Load-bearing premise

The argument only binds if, at the dialogue turns the policy actually practices, the simulated user almost always gives its single most-likely reply; if the simulator is genuinely varied exactly where the policy goes, the gradient-bias and entropy-collapse chain need not hold, and the paper's zero-variance diagnostic cannot rule that out because all-failure batches also produce zero variance.

Editorial extensions

If this is right

  • Single-simulator RL should not be read by its best-checkpoint number: the paper's curves show that number is a transient peak, and by end of training the policy falls back toward an untrained baseline.
  • Verbalized Sampling recovers most of the held-out gain (up to 9 points on $\tau^2$-bench) without retraining the simulator or the policy, because it restores within-simulator response diversity at every turn.
  • Co-Training and Population Co-Training push held-out success further (up to 14 points) and keep policy entropy in a healthy range, because the simulator's mode moves as training proceeds.
  • The mechanism also applies to symmetric cooperation: against a fixed partner, cross-play plateaus and then loses conversation turns to a short exploit, while self-play and population self-play continue improving.
  • The human study on $\tau^2$-bench and Persuasion for Good shows that the fixes' advantage over single-simulator RL transfers to real users, not only to the LLM evaluation panel.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the collapse mechanism is general, any single fixed LLM serving as the training environment—a grader, a verification model, or a reward model—should show the same peak-then-degrade signature; a cheap test is to track zero-variance batch fraction and policy entropy while training against a frozen judge, exactly the diagnostic the paper applies to user simulators.
  • The theory predicts a precise sufficient condition that the paper does not directly measure: $\epsilon_\phi$ on the training-visit distribution. Sampling simulator responses at the states actually reached by the policy and estimating their deviation from the modal response would settle when the bound binds and when a supposedly frozen simulator is diverse enough to be safe.
  • Because Co-Training's gains depend on a simulator reward that preserves cross-checkpoint variation, the paper's own ablation suggests a meta-learning extension: learn the simulator reward to maximize disagreement across checkpoints rather than hand-designing a variance-targeting curriculum.
  • Population size $K$ has an interior optimum (the $K{=}5$ sweep beats $K{=}1,3,10$), which the authors attribute to stale checkpoints diluting the signal; an adaptive curator that drops redundant checkpoints should extend the method to longer training runs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies multi-agent RL for human-AI interaction where a single frozen LLM simulates the user. It claims that such simulators are mode-collapsed, that policies trained against them overfit to the simulator's modal behavior, and that this causes poor transfer to unseen simulators and real users. The authors formalize 'simulator collapse' through a definition of per-turn collapse error and prove that the policy gradient is biased toward a deterministic mode-user objective (Theorem 3.2), that simulator-side reward variance vanishes (Lemma 3.3), and that policy mass concentrates geometrically onto a mode-exploit set (Corollary 3.5). They propose two fixes: Verbalized Sampling, which draws from a verbalized response distribution at inference, and Co-Training, which jointly trains the simulator and policy, optionally with a population of simulator checkpoints. Experiments on Persuasion for Good, tau2-bench, and CooperBench show that single-simulator RL's held-out performance peaks early and declines, while the two fixes improve held-out scores, and a pre-registered human study on two benchmarks shows gains on real users. The paper also releases the SCOPE framework.

Significance. If the central claims hold, this is an important result for the LLM-agent training community: it identifies a structural failure mode of the common single-frozen-simulator recipe and offers two practical mitigations with gains on three benchmarks and a human study. The paper is unusually honest about its limitations, explicitly flagging the one-sided diagnostic, the unverified reference-recovery assumption, and the task-specific simulator reward. It also contributes the SCOPE framework, which could be useful infrastructure. However, the significance of the causal claim depends on closing the gap between the theoretical mechanism and its empirical measurement: the paper does not directly estimate the per-turn simulator collapse error on the training-visit distribution, and the held-out evaluation panel includes the training simulators. The human study is a notable strength, as are the detailed ablations of pool size and simulator reward.

major comments (4)
  1. [3.1, 3.2, Fig. 19] The central causal chain from simulator collapse to policy collapse is not directly tested. Definition 3.1 and Eq. (4) define the per-turn collapse error epsilon_phi on the training-visit distribution, and Theorem 3.2's bound is informative only when the accumulated error is small. The only empirical support is the zero-variance batch fraction (Fig. 19), which the authors themselves call 'one-sided': Lemma 3.3 gives one direction, but zero-variance batches can also arise from all-success/all-failure groups or from the agent's own strategy collapse, as Eq. (6) shows. A competing explanation is that any fixed stochastic environment (even a diverse simulator) induces policy entropy collapse under group-relative normalization, producing the same OOD decline without simulator-side mode collapse. Since the paper's headline claim is that the failure is traced to simulator collapse, the manuscript should include a direct measurement of epsilon_phi (or of the simulator-side term in Eq. 6) at the histories visited by the trained policy. Without that, the causal story is a hypothesis supported only indirectly.
  2. [4.1, Table 1, Fig. 21] The main experimental tables report the best checkpoint over training, not the final or steady-state performance. The text acknowledges that RL (Single)'s held-out score is a transient peak that collapses back toward the untrained baseline by end of training (Fig. 21). Reporting best-checkpoint numbers (e.g., 46.1 vs 40.4 on tau2-Retail in Table 1) can conceal the very failure mode the paper claims to identify: if the policy systematically degenerates, the appropriate summary is end-of-training performance, area under the curve, or a clearly stated convention that peak performance is the quantity of interest. The current presentation overstates the case that single-simulator RL 'systematically fails to generalize' while simultaneously claiming the collapse is in the curve shape; these two statements need to be reconciled with a consistent reporting standard.
  3. [4.1, Table 7, C.4] The 'held-out panel' used for OOD evaluation includes the three training simulators (GPT-5-mini, Haiku-4.5, Gemini-3-Flash) as three of its six members. Evaluating on the training simulators means the panel is not held out in the standard sense, and the reported OOD numbers include in-distribution assessment. This could substantially inflate the apparent generalization of single-simulator RL, which is trained against one of those models. The paper should either remove the training simulators from the eval panel or report the seen/unseen breakdown prominently in the main text (the appendix reference is not sufficient for the headline OOD claim). The claim 'these results show failure to transfer to unseen simulators' requires an eval set that is truly unseen.
  4. [3.4, Proposition 3.7, Appendix F.5] The theoretical justification for Verbalized Sampling rests on the reference-recovery assumption D_TV(p_VS_phi, P) <= eta, which is precisely the property that Verbalized Sampling is designed to achieve and which the manuscript explicitly leaves unverified ('we defer quantitative behavior-coverage measurements to future work'). Proposition 3.7 is therefore conditional on an assumption that is the method's intended effect, not an established fact about the implemented system. The empirical gains from VS are encouraging, but the statement that 'VS recovers the reference-user gradient' is not supported by the current evidence. The authors should provide a quantitative test of reference recovery, e.g., comparing the empirical behavior-class distribution under VS with the distribution under the pre-RLHF reference or another specified target.
minor comments (5)
  1. [2, Eq. (2)] The group-relative advantage definition divides by sigma_R, which is zero when all trajectories in a group receive the same terminal reward; the paper mentions this boundary case but does not state how it is handled in the implementation (e.g., zeroing the advantage or skipping the update). Please clarify.
  2. [Figure 3 caption] The caption says 'three single-simulator REINFORCE runs' while the text and Table 6 describe a GRPO-style clipped surrogate with group-relative normalization. The terminology should be consistent throughout.
  3. [Table 1] The subscript is described as 'panel-std over six held-out simulators', but the table also includes the three training simulators in the panel; the wording should distinguish 'evaluation panel' from 'held-out panel'.
  4. [Abstract] The abstract reports improvements of 'up to 9%' and '14%'; it would be clearer to state whether these are absolute percentage points or relative improvements, and to specify the task and model size.
  5. [Appendix F.5] Proposition F.1 has a typo in the bound: the displayed inequality P_gamma(B|s) <= rho / (m lambda^{gamma-1}) should presumably have the factor lambda^{gamma-1} in the numerator or denominator, depending on the derivation; the current form is dimensionally inconsistent with the steps in the proof.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity: Proposition 3.7's reference-recovery premise is the unverified intended effect of Verbalized Sampling, imported from overlapping-author prior work; the central empirical results are independent.

  1. self citation load bearing [Section 3.4, Proposition 3.7; Appendix F.5]
    "Proposition 3.7 (Reference-gradient recovery under Verbalized Sampling). If D_TV(p^VS_phi(·|s,a^pi), P(·|s,a^pi)) ≤ eta(s,a^pi) ... then ... ||∇_θ J_VS(θ) − ∇_θ J_ref(θ)|| ≤ 2 B R_max \bar{eta}_H(θ). ... The closeness D_TV(p^VS_phi, P) ≤ eta is an empirical assumption; Appendix F.5 discusses when it holds and when it fails. ... The reference-recovery assumption ... is an empirical claim. ... we defer quantitative behavior-coverage measurements to future work."

    The proposition's premise (VS distribution is η-close to the reference distribution P) is exactly the property that Verbalized Sampling is claimed to achieve via [22], whose author list overlaps with the present paper (Yu, Chong, Manning, Shi). The conclusion (VS recovers the reference-user gradient) is therefore a restatement of the assumed effect rather than independently derived evidence, and the paper explicitly defers quantitative verification of the premise. This is load-bearing for the theoretical mechanism attributed to VS, though the empirical Tables 1 and 3 provide independent support for the headline results, so the circularity is minor.

full rationale

The paper's central claim—single-simulator RL collapses and VS/Co-Training recover—is validated empirically on three benchmarks and a human study, so it is not forced by construction. The theory is conditional: Theorem 3.2 and Corollary 3.5 are bounds that hold when the per-turn collapse error is small and the exploit set is fixed; the paper's own one-sided diagnostic is explicitly labeled not a direct estimator of epsilon_phi. The only circularity-adjacent step is Proposition 3.7, whose reference-recovery premise is the unverified intended effect of Verbalized Sampling, imported from overlapping-author prior work [22] and deferred to future work in Appendix F.5. Because the empirical results do not depend on that proposition's truth, the overall circularity score is 2.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central mechanism is conditional on mode collapse at visited histories, a fixed nonempty mode-exploit set, and a task-specific curriculum reward for Co-Training. The paper does not directly estimate epsilon or Delta_x; instead it uses a one-sided variance diagnostic and qualitative transcript inspections. The curves and human study make the qualitative claim plausible, but the headline numbers are measurements, not predictions from the theory.

free parameters (6)
  • Collapse threshold epsilon_star = not assigned
    Defines epsilon-star-collapsed in Definition 3.1; the theory is stated for any threshold, so this is a conceptual ordering parameter rather than a fitted constant.
  • Co-Training curriculum target within-batch variance = 0.25
    The simulator reward is shaped to keep within-batch variance near 0.25; the authors state this choice is essential, and Appendix F.8 shows both adversarial and cooperative rewards collapse the simulator onto a new mode.
  • Population pool size K = 5
    Swept over K in {1,3,5,10} and chosen as the best; K=5 and K=10 perform comparably, with K=5 slightly ahead on both P4G and tau2-Retail (Appendix F.1).
  • Verbalized Sampling candidate count K_VS = 5
    Number of verbalized response candidates sampled per simulator turn; taken from the prior VS method rather than swept here.
  • Policy learning rate = 1e-6
    Tuned on the first 50 steps of tau2-bench Ensemble training and then shared across all methods and tasks (Appendix C.4).
  • Group size G = 8
    Samples per prompt used for group-relative advantage normalization; this controls the within-batch reward contrast that the collapse theory relies on.
assumptions (6)
  • domain assumption Aligned LLM simulators are mode-collapsed at the histories visited during training (small per-turn collapse error).
    Load-bearing premise of Theorem 3.2 and Corollary 3.5; supported by external citations [20,27,22,21] and by the paper's zero-variance diagnostic, but epsilon is not directly estimated.
  • domain assumption Verbalized Sampling satisfies reference-recovery D_TV(pVS_phi, P) <= eta on visited states.
    Required by Proposition 3.7. The authors explicitly call this an empirical assumption that can fail and defer quantitative behavior-coverage tests to future work (Appendix F.5).
  • domain assumption The pre-RLHF reference distribution P is broader than the aligned simulator's mode and is the relevant target for VS.
    Proposition 3.7 recovers the reference-user gradient, not the real-user gradient; the paper notes P is distinct from P_real and uses the human study for the real-user step.
  • domain assumption The mode-exploit gap Delta_x > 0 exists on the strategy abstraction Y.
    Assumption B.3 is needed for Corollary 3.5's geometric concentration; the gap is never measured directly, only inferred from entropy collapse and transcript similarity.
  • domain assumption The curriculum simulator reward preserves informative variation across checkpoints.
    Corollary B.6 and the population benefit depend on simulator modes staying distinct; the paper states it has not mapped what other rewards would work (Appendix A.1).
  • domain assumption The idealized KL-regularized softmax update on strategy clusters approximates the token-level GRPO step.
    Proposition 3.4 is explicitly presented as a sufficient mechanism, not a direct theorem about the implemented optimizer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL." pith.science (2026). https://pith.science/paper/J7VUG42J

@misc{pith2026260812253,
  author       = {Pith},
  title        = {Pith review of: One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7VUG42J}},
  note         = {Machine review of arXiv:2608.12253}
}
abstract

Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

Figures

Figures reproduced from arXiv: 2608.12253 by the authors.

Figure 1
Figure 1. Single-simulator RL collapses; Verbalized Sampling and Co-Training recover. τ 2 - bench (Qwen3-4B-Instruct). (a) Held-out generalization: RL against a single frozen simulator peaks early, then starts collapsing. (b) Policy entropy: the same recipe drives the policy’s entropy to near zero. (c) Human study: both fixes lift real-user performance over RL (Single), which drops below even the untrained baseline. Preprint.… view at source ↗
Figure 2
Figure 2. Simulator collapse and our two fixes. (a) Problem: the real-user distribution is broad, but a frozen LLM simulator covers only one mode; the RL policy locks onto that mode and gives the narrow reply on real users it cannot serve. (b) Verbalized Sampling (inference-time): a single prompt asks the still-frozen simulator for several plausible user replies with likelihoods, so the policy sees varied reactions (acceptanc… view at source ↗
Figure 3
Figure 3. Single-simulator RL exhibits simulator collapse. Three single-simulator REINFORCE runs, three seeds each, ±1σ shading on OOD. Training reward climbs in every run (a), OOD eval peaks early and declines (b), and policy entropy collapses (c). The decoupling between (a) and (b) is simulator collapse passing through into policy collapse; (c) is the mechanism. Token-level entropy, which we measure in §3.3, is the empirica… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Both inference- and training-time solutions revive policy entropy. (a) P4G Eval Reward over training; (b) τ 2 -bench Retail Eval Success Rate; (c) τ 2 -bench Retail Policy Entropy. RL (Single) rises briefly and collapses below the untrained baseline on both eval panels…
Figure 5
Figure 5. Figure 5: Persuader (agent) system prompt for Persuasion for Good. The persona block is loaded [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Persuadee (user simulator) system prompt for Persuasion for Good. The structured [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Verbalized-sampling instruction block appended to the persuadee system prompt for the [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: τ 2 -bench user-simulator system prompt. The {instructions} placeholder is filled per-task with the scenario-specific instruction block from the retail or airline split [11]. CooperBench coding-agent system prompt You are a helpful assistant that can interact with a co…
Figure 9
Figure 9. Figure 9: Coding-agent system prompt for CooperBench (cooperative setting) [ [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: τ 2 -bench landing view. The right panel renders the structured task profile (goal, role information, conditional behaviors, style notes, and a sky-blue Travelers callout when the scenario involves multiple passengers). The left panel is a single-message-per-turn stre…
Figure 11
Figure 11. Figure 11: Persuasion for Good landing view. The right panel renders the persuadee role-play prompt drawn from the ConvoKit corpus, including the persona’s demographics, salient personality traits, and behavioural guidance (e.g., “Don’t agree or refuse on the first message”). Th…
Figure 12
Figure 12. Figure 12: τ 2 -bench survey panel. Rendered after the participant confirms the End modal. The eight 1–7 Likert items (task success, helpfulness, honesty, efficiency, instruction-following, safety, frustration, overall satisfaction) replace the instruction panel on the right; th…
Figure 13
Figure 13. Figure 13: Persuasion for Good survey panel. Differs from the τ 2 survey in two ways: the first item is a continuous donation amount in [$0, $2] rather than a Likert score, and the five Likerts measure perceptions of the persuader (argument quality, empathy, perceived manipulati…
Figure 14
Figure 14. Figure 14: Debrief page. Surfaces the Prolific completion code, an eight-second auto-redirect countdown back to Prolific, the post-hoc task-outcome verdict from the τ 2 evaluator, and a debrief block revealing which agent condition the participant interacted with (blinded during…
Figure 15
Figure 15. Figure 15: Human study on τ 2 -bench. Each subplot reports mean ±1 std across N=40 Prolific participants per condition. ∗ p<0.05, ∗∗ p<0.01 vs RL (Single) (Welch’s t, Holm–Bonferroni within each panel; 2 pre-registered pairwise tests). F.4 Per-benchmark training, eval, and entro…
Figure 16
Figure 16. Figure 16: Human study on Persuasion for Good. Each subplot reports mean ±1 std across N=40 Prolific participants per condition. ∗ p<0.05, ∗∗ p<0.01 vs RL (Single) (Welch’s t, Holm–Bonferroni within each panel; 2 pre-registered pairwise tests). lower training reward for a higher…
Figure 17
Figure 17. Figure 17: K=5 peaks above both extremes; the optimum is interior because stale checkpoints dilute the population. Eval reward over training for K ∈ {1, 3, 5, 10} on P4G (left) and τ 2 -Retail (right); checkpoint cadence is every four training steps. Asymptote ordering: K=5 > K=…
Figure 18
Figure 18. Figure 18: τ 2 -bench with Qwen3-8B. Scaling the trainable policy from Qwen3-4B-Instruct to Qwen3-8B preserves the ordering from [PITH_FULL_IMAGE:figures/full_fig_p035_18.png]
Figure 19
Figure 19. Figure 19: Empirical training dynamics on τ 2 -bench Retail (full time series). Zero-variance batch fraction (top-left), all-success (top-right), all-failure (bottom-left), and policy entropy (bottom-right). Single-simulator RL (blue) blows up on every panel; Co-Training and Pop…
Figure 20
Figure 20. Figure 20: Persuasion for Good: training, OOD eval, and policy entropy. Six methods, three seeds each; ±1σ shading on OOD. Training reward against the GPT-5-mini training simulator (a) is highest for RL (Single), which corresponds to the lowest agent entropy (c) and the worst OO…
Figure 21
Figure 21. Figure 21: τ 2 -bench Retail: training, OOD eval, and policy entropy. Seven methods, three seeds each. Same qualitative pattern as [PITH_FULL_IMAGE:figures/full_fig_p037_21.png]
Figure 22
Figure 22. Figure 22: τ 2 -bench Retail (Qwen3-8B). Six methods, three seeds each; ±1σ shading on OOD. The collapse pattern from [PITH_FULL_IMAGE:figures/full_fig_p037_22.png]
Figure 23
Figure 23. Figure 23: CooperBench (Qwen3.5-9B): training, eval, entropy, and conversation turns. The 9B run shows the same overfit signature as [PITH_FULL_IMAGE:figures/full_fig_p038_23.png]
Figure 24
Figure 24. Figure 24: CooperBench (Qwen3.5-27B): training, eval, entropy, and conversation turns. Cross￾play against a fixed Haiku partner or against a K=3 frozen ensemble shows the overfit signature on conversation turns: turns rise early as the policy learns to interact, then drop as a s…
Figure 25
Figure 25. Figure 25 [PITH_FULL_IMAGE:figures/full_fig_p039_25.png]
Figure 26
Figure 26. Figure 26: Verbalized Sampling against GPT-5. Three seeds per setting. (a) Training reward: RL (Single) climbs to a plateau in the 0.6–0.8 range; +VS is noisier and lower because the K-modal verbalized simulator gives less consistent reward signal. (b) OOD eval (held-out 6-model…
Figure 27
Figure 27. Figure 27: Co-Training requires a carefully chosen simulator reward (τ 2 -bench Retail). Top-left: agent training reward; cooperative reward (green) reaches the highest training reward by reward￾hacking a trivially helpful simulator. Top-right: opponent reward; only the curricul…
Figure 28
Figure 28. Figure 28: Persuasion for Good (Olmo-3-7B-Instruct). Three panels: training reward, OOD eval reward, policy entropy. The qualitative simulator-collapse signature reproduces: RL (Single) saturates training reward, peaks transiently on OOD, and crashes its (higher initial) entropy…
Figure 29
Figure 29. Figure 29: τ 2 -bench Retail (Olmo-3-7B-Instruct). Same three panels as [PITH_FULL_IMAGE:figures/full_fig_p041_29.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 20 canonical work pages

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Understanding r1-zero-like training: A critical perspective, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025. URL https: //arxiv.org/abs/2503.20783

    arXiv 2025

  3. [3]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    arXiv 2025

  4. [4]

    Retool: Reinforcement learning for strategic tool use in llms, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URLhttps://arxiv.org/abs/2504.11536

    arXiv 2025

  5. [5]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URLhttps://arxiv.org/abs/2503.09516

    arXiv 2025

  6. [6]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025. URLhttps://arxiv.org/abs/2504.21798

    arXiv 2025

  7. [7]

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution, 2025. URL https://arxiv.org/ abs/2502.18449. 11

    arXiv 2025

  8. [8]

    UserRL: Training Interactive User-Centric Agent via Reinforcement Learning, September

    Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. UserRL: Training Interactive User-Centric Agent via Reinforcement Learning, September

    work page

Show all 75 references
  1. [9]

    TOM-SWE: User Mental Modeling For Software Engineering Agents, October 2025

    Xuhui Zhou, Valerie Chen, Zora Zhiruo Wang, Graham Neubig, Maarten Sap, and Xingyao Wang. TOM-SWE: User Mental Modeling For Software Engineering Agents, October 2025. URLhttp://arxiv.org/abs/2510.21903. arXiv:2510.21903 [cs]

    2025

  2. [10]

    HumanLM: Simulating Users with State Alignment Beats Response Imitation, February 2026

    Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. HumanLM: Simulating Users with State Alignment Beats Response Imitation, February 2026. URL https://arxiv. org/abs/2603.03303

    2026

  3. [11]

    τ 2-Bench: Evaluating Conversational Agents in a Dual-Control Environment, June 2025

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2-Bench: Evaluating Conversational Agents in a Dual-Control Environment, June 2025. URL http: //arxiv.org/abs/2506.07982. arXiv:2506.07982 [cs]

    2025 arXiv

  4. [12]

    Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J

    Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K. Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J. Ryan, Jiaxin Pei, and Diyi Yang. CooperBench: Why Coding Agents Cannot be Your Teammates Yet, January 2026. URL http://arxiv. org/abs/2601.13295....

    2026

  5. [13]

    Position: Humans are missing from ai coding agent research

    Zora Zhiruo Wang, John Yang, Kilian Lieret, Alexa Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik Narasimhan, Ludwig Schmidt, Graham Neubig, Daniel Fried, and Diyi Yang. Position: Humans are missing from ai coding agent research. https: //zorazrw.gi...

    2026

  6. [14]

    Persuasion for good: Towards a personalized persuasive dialogue system for social good

    Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 56...

    2019

  7. [15]

    Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning, October

    Marwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff, Sergey Levine, and Natasha Jaques. Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning, October

  8. [16]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. InProceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New ...

    2023

  9. [17]

    arXiv:2511.00222 [cs]

    URLhttp://arxiv.org/abs/2511.00222. arXiv:2511.00222 [cs]

  10. [18]

    Sotopia-RL: Reward Design for Social Intelligence, October 2025

    Haofei Yu, Zhengyang Qi, Yining Zhao, Kolby Nottingham, Keyang Xuan, Bodhisattwa Prasad Majumder, Hao Zhu, Paul Pu Liang, and Jiaxuan You. Sotopia-RL: Reward Design for Social Intelligence, October 2025. URL http://arxiv.org/abs/2508.03905. arXiv:2508.03905 [cs]

    2025

  11. [19]

    Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S

    Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S. Bernstein. Llm agents grounded in self-reports enable general-purpose simulation of individuals, 2024. URLh...

    2024 arXiv

  12. [20]

    Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), October 2025

    Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), October 2025. URL http://arxiv.org/ abs/2510.22954. arXiv:2...

    2025

  13. [21]

    Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023

    Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023. URLhttps://arxiv.org/abs/2311.18232

    2023 arXiv

  14. [22]

    Tomz, Christopher D

    Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R. Tomz, Christopher D. Manning, and Weiyan Shi. Verbalized sampling: How to mitigate mode collapse and unlock llm diversity, 2025. URLhttps://arxiv.org/abs/2510.01171

    2025 arXiv

  15. [23]

    KL-regularized reinforcement learning is designed to mode collapse, 2025

    Anthony GX-Chen, Jatin Prakash, Jeff Guo, Rob Fergus, and Rajesh Ranganath. KL-regularized reinforcement learning is designed to mode collapse, 2025. URL https://arxiv.org/abs/ 2510.20817. 12

    2025

  16. [24]

    Chasing moving targets with online self-play reinforcement learning for safer language models, 2025

    Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models, 2025

    2025

  17. [25]

    Natural emergent misalignment from reward hacking in production RL, 2025

    Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas...

    2025

  18. [26]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine Learning, 8:229–256, 1992. doi: 10.1007/BF00992696

    1992 doi

  19. [27]

    SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning, July 2025

    Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning, July 2025. URL...

    2025

  20. [28]

    Flipping the Dialogue: Training and Evaluating User Language Models, October 2025

    Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. Flipping the Dialogue: Training and Evaluating User Language Models, October 2025. URL http://arxiv.org/abs/2510. 06552. arXiv:2510.06552 [cs] version: 1

    2025

  21. [29]

    Noveltybench: Evaluating language models for humanlike diversity,

    Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. Noveltybench: Evaluating language models for humanlike diversity,

  22. [30]

    URLhttps://arxiv.org/abs/2504.05228

    arXiv

  23. [31]

    SPICE: Self-Play In Corpus Environments Improves Reasoning, October 2025

    Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. SPICE: Self-Play In Corpus Environments Improves Reasoning, October 2025. URL http://arxiv.org/abs/2510. 24684. arXiv:2510.24684 [cs]

    2025

  24. [32]

    Mind the sim2real gap in user simulation for agentic tasks, 2026

    Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, and Maarten Sap. Mind the sim2real gap in user simulation for agentic tasks, 2026. URL https://arxiv.org/abs/2603.11245

    2026 arXiv

  25. [33]

    Measuring and mitigating the distributional gap between real and simulated user behaviors, May 2026

    Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar, Marwa Abdulhai, Sergey Levine, Michel Galley, and Dilek Hakkani-Tür. Measuring and mitigating the distributional gap between real and simulated user behaviors, May 2026. URLhttps://arxiv.org/abs/2605.07847

    2026 arXiv

  26. [34]

    Ragen-2: Reasoning collapse in agentic rl, 2026

    Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen-2: Reasoning collapse in agentic rl, 2026. URL https: //arxiv.org/abs/2...

    2026 arXiv

  27. [35]

    Hi- erarchical agenda reasoning for strategic multi-turn dialogue agents

    Marwa Abdulhai, Ryan Cheng, Aryansh Shrivastava, Aviral Kumar, and Sergey Levine. Hi- erarchical agenda reasoning for strategic multi-turn dialogue agents. InWorkshop on Scaling Post-training for LLMs, 2026. URLhttps://openreview.net/forum?id=p144zx4bO0

    2026

  28. [36]

    Llm probability concentration: How alignment shrinks the generative horizon, 2025

    Chenghao Yang, Sida Li, and Ari Holtzman. Llm probability concentration: How alignment shrinks the generative horizon, 2025. URLhttps://arxiv.org/abs/2506.17871

    2025

  29. [37]

    MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use, August 2025

    Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai. MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use, August 2025. URL http://arxiv.org/abs/2508.18669. arXiv:2508.18669 [cs]. 13

    2025 arXiv

  30. [38]

    Zhang, Chenghao Yang, Ningshan Ma, Weihao Xuan, and Jen tse Huang

    Yunze Xiao, Vivienne J. Zhang, Chenghao Yang, Ningshan Ma, Weihao Xuan, and Jen tse Huang. The chameleon’s limit: Investigating persona collapse and homogenization in large language models, 2026. URLhttps://arxiv.org/abs/2604.24698

    2026 arXiv

  31. [39]

    Richardson, Austin C

    Jacy Reese Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, James Evans, Erik Brynjolfsson, and Michael Bernstein. Llm social simulations are a promising research method, 2025. URLhttps://arxiv.org/abs/2504.02234

    2025 arXiv

  32. [40]

    Beyond cooperative simulators: Generating realistic user personas for robust evaluation of llm agents, 2026

    Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, and Natasha Jaques. Beyond cooperative simulators: Generating realistic user personas for robust evaluation of llm agents, 2026. URLhttps://arxiv.org/abs/2605.12894

    2026 arXiv

  33. [41]

    Training Proactive and Personalized LLM Agents, November 2025

    Weiwei Sun, Xuhui Zhou, Weihua Du, Xingyao Wang, Sean Welleck, Graham Neubig, Maarten Sap, and Yiming Yang. Training Proactive and Personalized LLM Agents, November 2025. URLhttp://arxiv.org/abs/2511.02208. arXiv:2511.02208 [cs]

    2025

  34. [42]

    Kanishk Gandhi, Agam Bhatia, and Noah D. Goodman. Learning to Simulate Human Dialogue, January 2026. URLhttp://arxiv.org/abs/2601.04436. arXiv:2601.04436 [cs]

    2026

  35. [43]

    Enhancing personalized multi-turn dialogue with curiosity reward, 2025

    Yanming Wan, Jiaxing Wu, Marwa Abdulhai, Lior Shani, and Natasha Jaques. Enhancing personalized multi-turn dialogue with curiosity reward, 2025. URL https://arxiv.org/ abs/2504.03206

    2025

  36. [44]

    Quantifying the utility of user simulators for building collaborative llm assistants, 2026

    Joseph Suh, Ayush Raj, Minwoo Kang, and Serina Chang. Quantifying the utility of user simulators for building collaborative llm assistants, 2026. URL https://arxiv.org/abs/ 2605.09808

    2026 arXiv

  37. [45]

    Non- collaborative user simulators for tool agents, September 2025

    Jeonghoon Shim, Woojung Song, Cheyon Jin, Seungwon KooK, and Yohan Jo. Non- collaborative user simulators for tool agents, September 2025. URL https://arxiv.org/ abs/2509.23124

    2025

  38. [46]

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raima...

    1912 arXiv

  39. [47]

    Bakker, and Jiaxin Pei

    Shu Yang, Shenzhe Zhu, Hao Zhu, José Ramón Enríquez, Di Wang, Alex Pentland, Michiel A. Bakker, and Jiaxin Pei. Multi-user large language model agents, 2026. URL https://arxiv. org/abs/2604.08567

    2026 arXiv

  40. [48]

    David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Made...

    2016 doi

  41. [49]

    Coevolving with the other you: Fine-tuning LLM with sequential cooperative multi-agent reinforcement learning

    Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning LLM with sequential cooperative multi-agent reinforcement learning. InAdvances in Neural Information Processing Systems, volume 37, pages 15497–15525, 2024

    2024

  42. [50]

    Absolute Zero: Reinforced Self- play Reasoning with Zero Data, October 2025

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shen- zhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute Zero: Reinforced Self- play Reasoning with Zero Data, October 2025. URL http://arxiv.org/abs/2505.03335. arXiv:2505.03335 [cs]

    2025 arXiv

  43. [51]

    Efficacy of language model self-play in non- zero-sum games, 2024

    Austen Liao, Nicholas Tomlin, and Dan Klein. Efficacy of language model self-play in non- zero-sum games, 2024. URLhttps://arxiv.org/abs/2406.18872

    2024 arXiv

  44. [52]

    SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks, March 2025

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks, March 2025. URLhttp://arxiv.org/abs/2503.15478. arXiv:2503.15478 [cs]

    2025 arXiv

  45. [53]

    Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, and Bo An. Dr. MAS: Stable reinforce- ment learning for multi-agent LLM systems, 2026

    2026

  46. [54]

    Tool-R0: Self-evolving LLM agents for tool-learning from zero data, 2026

    Emre Can Acikgoz, Cheng Qian, Jonas Hübotter, Heng Ji, Dilek Hakkani-Tür, and Gokhan Tur. Tool-R0: Self-evolving LLM agents for tool-learning from zero data, 2026

    2026

  47. [55]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  48. [56]

    Natural Language Actor- Critic: Scalable Off-Policy Learning in Language Space, December 2025

    Joey Hong, Kang Liu, Zhan Ling, Jiecao Chen, and Sergey Levine. Natural Language Actor- Critic: Scalable Off-Policy Learning in Language Space, December 2025. URL http:// arxiv.org/abs/2512.04601. arXiv:2512.04601 [cs]. 14

    2025

  49. [57]

    Ma, Seun Eisape, Ellie French, Tingting Du, Tianjiao Zhang, Alexander Koller, and Alane Suhr

    Nicholas Tomlin, Naitian Zhou, Eve Fleisig, Liangyuan Chen, Téa Wright, Lauren Vinh, Laura X. Ma, Seun Eisape, Ellie French, Tingting Du, Tianjiao Zhang, Alexander Koller, and Alane Suhr. Characterizing Language Use in a Collaborative Situated Game, December 2025. URLhttp://ar...

    2025

  50. [58]

    Openai gym.arXiv preprint arXiv:1606.01540, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym.arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  51. [59]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  52. [60]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  53. [61]

    slime: An llm post-training framework for rl scaling

    Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv

    2025

  54. [62]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs.arXiv preprint arXiv:2312.07104, 2024

    2024 arXiv

  55. [63]

    Megatron-lm: Training multi-billion parameter language models using model parallelism.CoRR, abs/1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.CoRR, abs/1909.08053, 2019. URLhttp://arxiv.org/abs/1909.08053

    1909 arXiv

  56. [64]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5

    2026

  57. [65]

    HybridFlow: A flexible and efficient RLHF framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. 2024

    2024

  58. [66]

    Morley Mao, Ion Stoica, Jiawei Zhao, and Beidi Chen

    Haizhong Zheng, Yizhuo Di, Jiahui Wang, Shuowei Jin, Xueshen Liu, Yongji Wu, Z. Morley Mao, Ion Stoica, Jiawei Zhao, and Beidi Chen. Astraflow: Dataflow-oriented reinforcement learning for agentic llms, 2026. URLhttps://arxiv.org/abs/2605.15565

    2026 arXiv

  59. [68]

    Team Olmo, :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heine- man, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, P...

    2025 arXiv

  60. [69]

    SPIRAL [ 25], SPICE [31], Absolute Zero [47]): one model serves both roles with role-specific loss masks

    Online self-play(cf. SPIRAL [ 25], SPICE [31], Absolute Zero [47]): one model serves both roles with role-specific loss masks

  61. [70]

    Co-Training(cf. Dr. MAS [ 50]): two separate models trained simultaneously on their respective turns of the same conversation

  62. [71]

    Updated?

    Co-Training with opponent pool(ours): Co-Training augmented with a checkpoint pool P of historical opponent snapshots; each rollout loads opponent weights from a pool-sampled checkpoint, and GRPO’s clipped importance ratio corrects for the off-policy gap. Paradigm coverage ver...

  63. [72]

    No training Base — — — —

  64. [73]

    Single-simulator (frozen) RL (Single) GPT-5-mini (single frozen LLM) No Single response Frozen single LLM Persona-Guided GPT-5-mini + persona prompt No One persona- conditioned response Prompt-only widening Verbalized Sampling GPT-5-mini No Sample one ofkverbal- ized candidate...

  65. [74]

    Multi-simulator (frozen) Ensemble Models (K=3){Haiku 4.5, GPT-5-mini, Gemini 3 Flash} No Cyclic rotation across rollouts Cross-family heterogene- ity

  66. [75]

    responses

    Trainable simulator (ours) Co-Training 1 trainable LLM Yes Current weightsϕ (t) Simulator adaptivity Population Co-TrainingFIFO buffer ofK=5his- toricalϕcheckpoints Yes Uniform sample from buffer Diversity+adaptivity C.4 Hyperparameters All methods (RL Single, Verbalized Sampl...

  67. [2025]

    arXiv:2509.19736 [cs]

    URLhttp://arxiv.org/abs/2509.19736. arXiv:2509.19736 [cs]

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.