Pith. sign in

REVIEW 1 major objections 1 minor 4 references

Multi-agent RL improves LLM workflows depending jointly on workflow, task, and scale rather than policy sharing alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 15:43 UTC pith:XXKSVFJ2

load-bearing objection The paper maps multi-agent RL tradeoffs across workflows and scales with a gradient-dynamics account, but that account looks interpretive rather than directly tested. the 1 major comments →

arxiv 2605.24202 v2 pith:XXKSVFJ2 submitted 2026-05-22 cs.AI cs.LG

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

Yifan Zeng , Yiran Wu , Yaolun Zhang , Wentian Zhao , Kun Wan , Qingyun Wu , Huazheng Wang This is my paper
classification cs.AI cs.LG
keywords multi-agent RLLLM workflowspolicy sharinggradient dynamicsworkflow topologyreinforcement learningagent rolesaccuracy degradation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that applying reinforcement learning to train roles in multi-agent LLM systems typically raises accuracy above base models, yet the size of the gains and the stability of training vary with the workflow structure, the nature of the task, and the size of the underlying models. Isolated-Policy training, where each role has separate parameters, often achieves the highest accuracies but is more likely to encounter a sharp drop in performance late in training. Shared-Policy training, by contrast, avoids some of those drops but produces different kinds of failures because the single policy gets pulled toward the role that receives the most gradient updates. The authors link these outcomes to how the workflow's routing of prompts and the sharing of parameters shape the flow of gradients during training. This matters for practitioners who want to know whether investing in multi-agent RL will pay off for their particular setup.

Core claim

End-to-end RL training of multi-agent LLM workflows improves over base models in a manner that depends jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy training tends to reach higher peak accuracy yet falls off a terminal accuracy cliff more often because parallel same-role agents on shared prompts amplify per-role gradients. Shared-Policy training does not eliminate failure but redistributes it into qualitatively different patterns because asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role. These patterns are explained by role-level gradient dynamics induced by workflow topology and policy routing.

What carries the argument

Role-level gradient dynamics induced by workflow topology and policy routing, which control how training pressure is distributed across roles in Shared-Policy versus Isolated-Policy setups.

Load-bearing premise

That the tested combinations of workflows, tasks, and scales capture the general conditions under which multi-agent RL succeeds or fails and that the gradient dynamics explain the failures.

What would settle it

A new experiment at an untested scale or task where Isolated-Policy no longer shows terminal cliffs or where Shared-Policy produces no distinct failure patterns would falsify the dependence on workflow, task, and scale.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-agent RL gains vary by workflow (Eval-Opt, Voting, Orch-Workers), so different topologies produce different improvement levels.
  • Isolated-Policy amplifies gradients in parallel roles, leading to terminal degradation in Voting and Orch-Workers.
  • Shared-Policy causes capture by the dominant role due to asymmetric gradient mass, producing task-specific failure signatures.
  • These effects appear across math and code tasks and model scales from 0.6B to 4B.
  • Policy sharing routes training pressure differently rather than providing uniform stability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Practitioners should evaluate both policy configurations at their target scale and task instead of assuming shared policies are always more stable.
  • The gradient mechanisms imply that altering prompt routing or adding role-specific regularization could reduce the observed failure modes.
  • Similar tradeoffs may appear in non-LLM multi-agent RL settings where roles interact through shared environments.
  • Future experiments at larger scales could test whether the scale dependence persists or if failures become less pronounced.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper empirically studies when end-to-end RL training of multi-agent LLM workflows improves over base models. It compares Shared-Policy (single policy for all roles) versus Isolated-Policy (per-role parameters) training across Eval-Opt, Voting, and Orch-Workers workflows on math and code tasks at three scales (0.6B, 1.7B, 4B). Key claims are that gains depend jointly on workflow, task, and scale rather than policy sharing alone; Isolated-Policy reaches higher peak accuracy but more often exhibits terminal accuracy cliffs, while Shared-Policy redistributes failures into different patterns; and these patterns are explained by role-level gradient dynamics induced by workflow topology and policy routing (parallel amplification under Isolated-Policy; asymmetric per-step mass under Shared-Policy).

Significance. If the empirical map and mechanistic account hold, the work is significant for mapping conditional tradeoffs in multi-agent RL for LLMs and for attempting to link observed accuracy patterns to workflow-induced gradient effects. The experimental matrix spanning multiple workflows, tasks, and scales is a clear strength, as is the focus on failure modes rather than only success cases. This could guide design choices in multi-agent LLM systems.

major comments (1)
  1. [gradient-dynamics explanation (abstract and analysis section)] The mechanistic explanation in the abstract (role-level gradient dynamics from workflow topology and policy routing, with parallel same-role agents amplifying gradients under Isolated-Policy and asymmetric per-step mass under Shared-Policy) is load-bearing for the central claim that gains 'depend jointly on workflow, task, and scale, not on policy sharing alone.' No indication is given that per-role gradient norms were measured, that ablations decoupled topology from reward sparsity or prompt overlap, or that gradient dynamics were logged; without these the account remains post-hoc interpretation of accuracy curves rather than a tested causal link.
minor comments (1)
  1. [abstract] The abstract states that multi-agent RL 'usually improves over base models' but provides no quantitative effect sizes, error bars, or details on how 'improvement' and 'failure' are operationalized across the experimental matrix.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and for highlighting both the strengths of the experimental matrix and the need for precision around the mechanistic account. We address the major comment below.

read point-by-point responses

  1. Referee: The mechanistic explanation in the abstract (role-level gradient dynamics from workflow topology and policy routing, with parallel same-role agents amplifying gradients under Isolated-Policy and asymmetric per-step mass under Shared-Policy) is load-bearing for the central claim that gains 'depend jointly on workflow, task, and scale, not on policy sharing alone.' No indication is given that per-role gradient norms were measured, that ablations decoupled topology from reward sparsity or prompt overlap, or that gradient dynamics were logged; without these the account remains post-hoc interpretation of accuracy curves rather than a tested causal link.

    Authors: We agree the gradient-dynamics discussion is interpretive rather than directly tested. The manuscript reports no per-role gradient norm measurements, no ablations separating topology from reward sparsity or prompt overlap, and no logged gradient trajectories. The core empirical claim—that performance gains depend jointly on workflow, task, and scale—is established by the accuracy results across the full matrix of workflows (Eval-Opt, Voting, Orch-Workers), tasks (math, code), and scales (0.6B–4B); this evidence stands independently of the proposed mechanism. The role-level gradient account is presented as a post-hoc explanation that aligns with the observed patterns of terminal cliffs under Isolated-Policy and redistributed failures under Shared-Policy. We will revise the abstract and analysis section to (i) explicitly label the gradient dynamics as an interpretive hypothesis consistent with the accuracy curves and workflow structure, (ii) separate it from the empirical claims, and (iii) note the absence of direct gradient or ablation evidence. This change will be made in the next revision. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical study with independent experimental observations

full rationale

The paper presents an empirical matrix of experiments across workflows (Eval-Opt, Voting, Orch-Workers), tasks (math, code), and scales (0.6B-4B), comparing Shared-Policy vs Isolated-Policy RL. Claims about joint dependence on workflow/task/scale and role-level gradient dynamics are interpretive summaries of observed accuracy curves, not reductions of predictions to fitted inputs or self-definitions by construction. No equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The gradient-dynamics account is post-hoc interpretation rather than a self-referential derivation, satisfying the criteria for a self-contained empirical result (score 0-2).

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is an empirical study relying on standard assumptions from reinforcement learning and LLM training literature rather than introducing new free parameters, axioms, or entities.

axioms (1)
  • domain assumption End-to-end reinforcement learning can be applied to optimize multi-agent LLM workflows for end-task accuracy.
    The study assumes this setup is feasible and examines when it yields improvements.

pith-pipeline@v0.9.1-grok · 5837 in / 1346 out tokens · 55790 ms · 2026-06-30T15:43:28.643240+00:00 · methodology

0 comments
read the original abstract

Multi-agent LLM workflows route inference through specialized roles to lift end-task accuracy, but jointly training those roles with reinforcement learning is unstable in ways that are poorly understood. We study when end-to-end RL training of multi-agent LLM workflows improves over their base models, comparing Shared-Policy training, where all roles update one policy, with Isolated-Policy training, where each role has its own parameters. Our experimental matrix spans Eval-Opt, Voting, and Orch-Workers workflows, math and code tasks, and three model scales (0.6B, 1.7B, 4B). We find that multi-agent RL usually improves over base models, but gains depend jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy tends to reach higher peak accuracy yet more often falls off a terminal accuracy cliff, while Shared-Policy training does not eliminate failure; it redistributes failure into qualitatively different patterns. We then explain the strongest of these patterns through role-level gradient dynamics induced by workflow topology and policy routing: under Isolated-Policy, parallel same-role agents on shared prompts amplify per-role gradients and drive terminal degradation in Voting and Orch-Workers workflows; under Shared-Policy, asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role, producing different failure signatures by task and workflow. Together, the empirical map and its underlying mechanisms show that policy sharing routes training pressure through different channels rather than offering uniform stability, making it a design choice with workflow- and task-conditional tradeoffs.

Figures

Figures reproduced from arXiv: 2605.24202 by Huazheng Wang, Kun Wan, Qingyun Wu, Wentian Zhao, Yaolun Zhang, Yifan Zeng, Yiran Wu.

Figure 1
Figure 1. Figure 1: Workflow topology and policy routing. Three workflows (Eval-Opt, Voting, Orch￾Workers), each trained under Isolated-Policy (one πrole per role) or Shared-Policy (one πshared for all roles), with a shared outcome-reward GRPO loop. Recent work has begun extending Group Relative Policy Optimization (Shao et al., 2024) to multi￾agent settings (Zhao et al., 2025b; Liu et al., 2025; Hong et al., 2025; Chen et al… view at source ↗
Figure 2
Figure 2. Figure 2: Per-cell validation accuracy delta against the base model, across the workflow [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: IP vs SP validation accuracy on matched cells. Markers above the diagonal favor IP. Marker shape encodes workflow, fill color encodes task (Math vs Code), and marker size encodes scale. Across the workflow, scale, and task matrix, the headline pattern is positive: multi-agent RL training reaches higher validation accuracy than the corresponding base model in the large majority of cells, and on most cells t… view at source ↗
Figure 4
Figure 4. Figure 4: Training-side dynamics at 1.7B × Math across the three multi-agent workflows (Eval-Opt, Voting, Orch-Workers). Top row: training success rate. Bottom row: per-checkpoint validation accuracy. accuracy. The matched comparison is summarized in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training-side instability amplitude on matched [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics on cliff cells. Each panel pairs a training-side amplitude diagnostic [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

    work page

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

    work page

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

    work page

  4. [4]

    We use upper-cased letter, e.g., X or X , to denote random vectors/variables, and lower-cased letter, e.g., x or x , to denote deterministic vectors/scalars

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page arXiv 2026