Machine Learning · arXiv:2602.08847 v2 · submitted 9 Feb 2026
Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems
The paper says
“MAS achieves clear gains over vanilla GRPO (e.g., +5.6% avg@16 and +4.6% pass@16 on math, and +15.2% avg@16 and +13.1% pass@16 on search) while largely eliminating gradient spikes.”
from the abstract · picked out as the main claim by a classifier
Reviews
Reviews by people
No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6% avg@16 and +4.6% pass@16 on math, and +15.2% avg@16 and +13.1% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-29 · no language-model review
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.09 · human participants 0.03 · controlled comparison 0.95 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. | background 0.88 | 0.06 | 0.03 | asserted 1.00 |
| In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. | contribution 1.00 | 0.92 | 0.07 | asserted 1.00 |
| We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. | result 0.73 | 0.90 | 0.15 | asserted 0.65 |
| Based on this finding, we propose Dr. | contribution 0.87 | 0.53 | 0.04 | unclear 0.77 |
| MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. | contribution 0.95 | 0.60 | 0.09 | asserted 0.93 |
| MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. | method 0.90 | 0.94 | 0.55 | asserted 1.00 |
| Beyond the algorithm, Dr. | contribution 0.88 | 0.58 | 0.03 | unclear 0.87 |
| MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. | contribution 0.61 | 0.83 | 0.06 | asserted 1.00 |
| MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. | method 0.93 | 0.87 | 0.04 | asserted 0.63 |
| MAS achieves clear gains over vanilla GRPO (e.g., +5.6\% avg@16 and +4.6\% pass@16 on math, and +15.2\% avg@16 and +13.1\% pass@16 on search) while largely eliminating gradient spikes. | result 1.00 | 0.96 | 0.45 | asserted 1.00 |
| Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency. | result 0.99 | 0.90 | 0.13 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.94
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-29: 1 request, 8,209 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 5 Pith papers
Reviewed papers on Pith that cite this one. OpenAlex reports about 20 citations worldwide. The full citation record
- Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation cs.AI · Nov 2026
- Collaborative Learning and Reasoning over Model Slices cs.AI · Oct 2026
- Train Locally, Learn Together: Distributed Multi-Agent Training via ADMM cs.AI · Oct 2026
- Belief-Conditioned Multi-Agent Actor Critic for Decentralized LLM Collaboration cs.AI · Oct 2026
- A Survey on RLOps for Large Language Models cs.SE · Nov 2026
How the literature uses it
9 arXiv papers cite this in 24 sentences
Papers citing it · 16 sentences without a confident label
Compares with it 1 paper
-
MAS [ref] and Stronger MAS [ref] build this support on top of verl [ref] , requiring pipeline-level modifications to coordinate multiple roles, policies, trainers, and weight streams, while also inheriting verl's colocated synchronous execution.
AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs · Application I: Multi-Policy Collaborative Training · 0.81
Cites it as support 2 papers
-
MAS [ref] documents that naïve GRPO becomes unstable at multi-agent scale under exactly this regime.
Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces · Trace length and credit fragility · 1.00
-
A global baseline $R̅ = 1/K∑_k R_task(τ_k)$ conflates reward distributions across roles with different execution contexts, which can obscure role-specific failures and produce poorly scaled advantage estimates [ref] .
GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence · Role-Contract Expectation Alignment (RCEA) · 0.97
Cites it as background 3 papers
-
The resulting long-horizon trajectories have also motivated work on training stability, credit assignment, and role-specific advantage estimation [ref] .
EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning · Agentic Reinforcement Learning · 0.88
-
Beyond single-agent optimization, researchers have started to explore reinforcement learning for multi-agent or multi-role systems, including different agent architectures and role-specific training objectives [ref] .
ICRL: Learning to Internalize Self-Critique with Reinforcement Learning · Related Work · 0.91
-
MAS [ref] identifies gradient-norm imbalance caused by global advantage normalization across agents and proposes agent-wise normalization.
When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs · Related Work · 0.89
Challenges and discussion
Know something about this paper? Leave a comment.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment