Pith. sign in

REVIEW 4 major objections 3 minor 12 references

For cooperative multi-agent PPO, this paper proves the expected gradient sees only the product of the two aggregation choices, and that variance makes per-agent ratios with neighborhood-sized advantages the strictly better design.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:35 UTC pith:D222UKNN

load-bearing objection New two-support formulation and a clean on-policy canonical form, but the off-policy dominance claim is unsupported and the gap is load-bearing. the 4 major comments →

arxiv 2607.17924 v1 pith:D222UKNN submitted 2026-07-20 cs.MA cs.LG

Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization

Zijian Zhao , Sen Li This is my paper
classification cs.MA cs.LG
keywords multi-agent reinforcement learningPPOadvantage supportimportance ratiovariance orderingcooperative MARLbias-variance tradeoffsupport matrix factorization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper isolates two knobs in cooperative multi-agent policy optimization: which agents' rewards enter an agent's advantage signal, and which agents' likelihood ratios are multiplied into its clipped importance weight. The authors prove a canonical structure — the expected policy gradient depends on the pair of choices only through their matrix product — so the two knobs are interchangeable with respect to the signal. The symmetry is broken by variance: rewards aggregate additively, while likelihood ratios aggregate multiplicatively, so a joint ratio's estimator variance grows exponentially with the number of agents. The resulting design rule is to keep the ratio per-agent and place all cross-agent aggregation in the advantage, sized to the true coupling neighborhood. If correct, this explains why local advantages succeed in practice while joint ratios are fragile, and it turns a hyperparameter choice into a modeling choice.

Core claim

At the behavior policy, the expected multi-agent PPO gradient depends on the advantage support S_A and ratio support S_R only through their matrix product S = S_R S_A (Theorem 1). Cross-agent importance ratios are therefore redundant: any aggregation placed in the ratio can be moved into the advantage as a linear reweighting with the same expected gradient. Variance breaks the tie: an advantage aggregates rewards as a sum with gentle variance growth, while a ratio aggregates likelihood ratios as a product whose variance factorizes into per-agent (1 + χ²_j) factors and grows exponentially with support size (Lemma 2). Among support pairs realizing a target product, the per-agent-ratio realizat

What carries the argument

The central objects are the two 0/1 support matrices — the advantage support S_A, which picks which agents' rewards form each agent's credit signal, and the ratio support S_R, which picks which agents' per-action likelihood ratios are multiplied into each agent's importance weight — together with their matrix product S = S_R S_A, the canonical form through which the expected gradient factorizes at the on-policy point (Theorem 1, proved via the score-function identity and the conditional independence of actions under a factorized policy). The companion mechanism is Lemma 2's variance factorization, which turns the variance of the ratio product into a product of (1 + χ²_j) factors, one per agg

Load-bearing premise

The load-bearing premise is that the support-product factorization of Theorem 1 — proved at θ = θ_old, where every likelihood ratio equals one — controls the comparison between the two aggregation paths when updates actually move off-policy (Section 3.4.3 and Appendix B.5 invoke Theorem 1 to assert the two paths share the same mean even at θ ≠ θ_old). If the ratio path's importance weights shift the expected gradient away from the advantage path's, then the universal rule 'ag

What would settle it

Compute, in a two-agent one-step game with coupled rewards, the exact off-policy expected gradient along the ratio path, E_{π_old}[ϱ_j r_j ∇_{θ_m} log π_m(a_m)] with ϱ_j = π_θ/π_old ≠ 1, and compare it with the advantage-path expectation E_{π_old}[r_j ∇_{θ_m} log π_m]. The paper's rule requires these to be equal because the two paths realize the same product S; any nonzero difference — which appears as soon as ϱ_j is correlated with the score–reward product — shows the equality of means holds only on-policy and that the Proposition 1 variance comparison is between estimators with different bia

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Corollary 1: joint and compound ratios add no expected-gradient signal beyond a linear reweighting of the advantage; every support pair is gradient-equivalent to a per-agent ratio with advantage S_R S_A.
  • Corollary 2 / Proposition 1: among realizations of a fixed product, the per-agent-ratio choice weakly dominates on-policy and strictly dominates off-policy in estimator variance, so a joint ratio should be replaced by the corresponding advantage reweighting.
  • Proposition 2: with the ratio per-agent, the advantage support has a bias–variance tradeoff with an interior MSE optimum exactly at the coupling neighborhood — smaller supports are biased, larger ones inflate variance.
  • The ratio's variance penalty is gated by the per-update policy shift χ²: under conservative trust regions it is dormant even at hundreds of agents, and it becomes the dominant failure mode when updates move off-policy, so step size, not agent count alone, is the trigger.
  • The rule transfers to centralized multi-action PPO built on a factorized joint policy: clip each component's ratio separately instead of as one n-way product, at no cost to the expected gradient.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the canonical form suggests a data-driven route to adaptive support when the coupling graph is unknown — estimate which neighbors' rewards outweigh their variance contribution per state, and realize the learned support on the advantage side only; the paper names this as future work but does not establish a procedure.
  • Testable extension: the clip-fraction diagnostic — joint-ratio clipping rising with agent count while per-agent clipping stays near zero — could serve as a deployment-time warning signal in large multi-agent systems, since the paper shows this pattern is the fingerprint of multiplicative variance rather than of any signal difference.
  • Neighbouring-problem connection: the results sharpen the exponential-decay locality arguments for networked multi-agent RL by separating the bias boundary (set by the coupling graph) from the variance cost (set by the support product), so systems with known interaction graphs can size advantages from the graph rather than by tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies two design choices in cooperative multi-agent PPO: which rewards enter each agent's advantage (support S_A) and which per-agent likelihood ratios enter its clipped importance weight (support S_R). It formalizes these as 0/1 support matrices and claims a canonical form: the expected policy-gradient signal depends on the pair only through the matrix product S_R S_A, so cross-agent ratios are redundant for the expected signal. Variance, however, breaks the tie because advantage aggregation is additive while ratio products have multiplicative, exponentially growing variance. This yields the design rule to aggregate advantages over the coupling neighborhood and keep ratios per-agent. The authors prove Theorem 1 at θ=θ_old, prove the ratio-moment lemmas, and validate predictions on four synthetic cooperative games and a 196-intersection traffic-signal control task, with code released.

Significance. If the full claim held, this would be a valuable parameter-free structural explanation for why neighbor-based advantages are common while local ratios are rare, with a direct design rule and empirical support. The paper's on-policy factorization (Theorem 1) is cleanly proven under explicit assumptions; Lemma 2 gives a sharp multiplicative-variance formula; and the experiments are carefully designed with known coupling graphs, including a real traffic-control validation with released code. The main weakness is that the headline universal rule is built on an off-policy mean-equality assertion that Theorem 1 does not establish; the paper's own Remark 1 identifies the off-policy regime as the one where ratios activate. The central design claims are therefore not fully supported as stated and need scoping or additional proof.

major comments (4)
  1. [§3.4.3, Proposition 1; Eq. (4); Theorem 1] Theorem 1 proves the support-product factorization only at θ=θold. Proposition 1, however, invokes it to assert that paths P and Q 'realize the same expected gradient by Theorem 1' when comparing variances for θ≠θold. At θ≠θold, Eq. (4) gives g_m^P = E[ϱ_m Σ_{j∈∂m} r_j s_m] while g_m^Q = Σ_{i∈∂m} E[(Π_{j∈∂i}ϱ_j) r_i s_m]; these coincide only under extra conditions not stated. Since PPO's inner epochs operate at θ≠θold, Corollary 2's 'strictly dominated off-policy' is not established, and a variance lower bound for Q does not imply MSE dominance when the means differ. Remark 1 concedes that the ratio activates only off-policy, the exact regime where Theorem 1 is silent. This gap must be closed or the universal design rule must be restricted to the on-policy/first-order regime.
  2. [Abstract; Corollary 1] The abstract and Corollary 1 state without qualification that the 'expected multi-agent policy optimization objective' depends on (S_A,S_R) only through S_R S_A. Theorem 1 proves this for the expected gradient evaluated at θ=θold, not for the objective value: at θ=θold, L(θold)=Σ_i(Σ_j S_A_ij E[r_j] − b_i), which depends on S_A alone. The claimed 'redundancy' is a statement about the on-policy gradient, and the text should say so wherever it appears. This is not merely a wording issue: an unqualified statement is false and can mislead practitioners comparing IPPO/MAPPO/joint-ratio variants.
  3. [Appendix B.6, Proposition 2(ii)] The proof that adding an uncoupled agent j to the advantage support 'adds a nonnegative variance contribution' is incomplete. The actual variance difference is Var(r_j s_m) + 2 Cov(r_j s_m, Σ_{k≠j} r_k s_m). The lower bound E[r_j^2 s_m^2]≥0 is only one term; the covariance can be negative and can dominate, so the total need not be nonnegative. Consequently the exact claim that the MSE-optimal advantage support is precisely the coupling neighborhood is not proven in general. Additional structure (for example nonnegative dependence or a conditioning argument separating r_j from the score) is needed, or the proposition should be weakened to a variance bound.
  4. [Appendix B.5, Eq. (15)] The lower bound for the ratio path's second moment is stated as E[(Π_{j∈∂i}ϱ_j)^2 r_i^2] ≥ δ E[(Π ϱ_j)^2], justified by conditional nondegeneracy of r_i. This is not valid as written: conditioning on a_m leaves r_i dependent on other action components, and the product includes ϱ_i, which is correlated with r_i. A lower bound on the full expectation cannot be obtained from a lower bound that holds only on a set of positive probability unless the complement's contribution is controlled. The variance domination may be true, but the proof needs a correct argument (e.g., summing over a subset of actions with positive πθold mass and using finiteness of the action set).
minor comments (3)
  1. [Section 3.4.2 / Appendix B.4] The term 'admissible' for A'_m = Σ eS_mj r_j should be clarified: eS entries are nonnegative integers, so A'_m is an integer-weighted reward sum. It would help to state explicitly how the baseline b_i is adjusted under this reweighting.
  2. [Remark 3] The claim that clipping 'cannot reverse Proposition 1' is stated without proof. Since clipping truncates the heavy-tailed product, it can reduce variance in some regimes; the argument that it cannot overturn the ordering needs a formal statement or a softened heuristic.
  3. [Notation] The phrase 'depends only on the product' can be misread as elementwise product; the text should explicitly say 'matrix product' in the abstract and introduction.

Circularity Check

1 steps flagged

No load-bearing circularity: Theorem 1 is a self-contained algebraic identity; the only reduction-by-construction is the Fig. 1(c) 'bias tracks together' illustration. The off-policy extension is a correctness gap, not circularity.

specific steps
  1. self definitional [Section 1, Figure 1 preview (panels c,d); cf. paths (7) in Section 3.4.3]
    "Panels (c,d) isolate this: they fix a matched effective support (so both realize the same expected gradient) and compare the two ways of reaching it—aggregating through the advantage (path P) versus through the ratio (path Q)—plotting their bias (c) and variance (d) as the support grows; the bias tracks together while the variance separates by up to 9×."

    The matched-effective-support construction fixes the two estimators to have the same expected gradient, and squared bias depends only on the estimator mean. Therefore equal bias is forced before any measurement; Fig. 1(c) is not an independent test of the redundancy prediction (R) but a restatement of the construction. This is a minor validation tautology and does not support the central theorem, which is derived algebraically from Eq. (4).

full rationale

The central derivation is self-contained. Theorem 1 evaluates the surrogate gradient (4) at θ=θold (where every ϱ_j=1 and w_i=1), uses the score-function identity and coupling to obtain g_m = Σ_{j:Cjm=1} (S_R S_A)_{mj} E[r_j ∇ log π_m]; no fitted parameter or external result is imported. Lemma 2 is a direct factorization of second moments under Assumptions 1–2. The paper's self-citations (Zhao et al., 2026; Zhao & Li, 2026) are used only as named examples of centralized/joint-ratio reductions, not as evidence for the theorem or the variance ordering, so they are not load-bearing. The only construction-reduces-to-input step is the Fig. 1(c) 'bias tracks together' illustration, which is forced by the matched-effective-support setup; it does not affect the theorem. A separate correctness caveat must be flagged: Remark 1 concedes Theorem 1 is on-policy, yet Proposition 1's proof (Appendix B.5) invokes Theorem 1 to assert equal means for θ≠θold. This is an unsupported extrapolation—the off-policy means of paths P and Q generally differ—so the 'strictly dominated off-policy' wording of Corollary 2 is not established. That is a validity gap, not circularity: the design rule is not equivalent to its inputs; it over-claims a theorem. Score 2 reflects the minor self-definitional validation and the self-citations, while the central claim retains independent mathematical content.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The theoretical derivation relies on four explicit domain assumptions plus an on-policy evaluation point; no fitted constants enter the central claims. The support matrices SA and SR are formal definitions, not newly postulated physical entities. Experimental hyperparameters (learning rate, batch size, reward weights) are configuration choices outside the derivation.

axioms (6)
  • domain assumption Factorized joint policy (Assumption 1): actions are conditionally independent across agents; joint policy factorizes as πθ(a)=∏_i πi_θ(ai).
    Required for the score-function factorization in Theorem 1 and for Lemma 2's variance product; fails for autoregressive action heads, which the paper excludes.
  • domain assumption Full support and finite second moments (Assumption 2): πi_θ(a)>0 for all i,a, and E[ϱj^2]<∞.
    Needed for importance-weighted estimators to have unit mean and finite variance; standard for softmax PPO.
  • domain assumption Symmetric coupling C with Cii=1 (Assumption 3): ri depends on aj iff Cij=1.
    Used to zero out E[r_j s_m] for non-coupled pairs and to state the coupling radius cleanly; directional case is only sketched in Remark 5.
  • domain assumption Bounded rewards (Assumption 4): |ri(a)|≤rmax<∞.
    Ensures advantage-side aggregation has variance O(|S_A|) rather than exponential; standard for bounded rewards.
  • ad hoc to paper On-policy evaluation θ=θold for gradient equivalence (Section 3.4.1).
    The redundancy theorem is proven only at θ=θold; the paper extends conclusions to off-policy updates without proving mean equivalence.
  • ad hoc to paper Clipping is 1-Lipschitz and does not reverse variance domination (Remark 3).
    Used to extend the variance comparison from unclipped to clipped PPO surrogates; stated as a remark rather than proved formally.

pith-pipeline@v1.3.0-alltime-deepseek · 20851 in / 13612 out tokens · 122777 ms · 2026-08-01T16:35:52.687497+00:00 · methodology

0 comments
read the original abstract

Multi-agent policy optimization, exemplified by PPO-based methods, is a key branch of cooperative Multi-Agent Reinforcement Learning (MARL). A central design question is how many neighboring agents\footnote{In this paper, "neighbors" refer not only to physical proximity but also to agents whose actions influence one another.} to aggregate in order to effectively utilize global information for cooperation. This decision must be made along two dimensions: in the advantage (which agents' rewards contribute to the credit signal) and in the ratio (which agents' likelihood ratios form the clipped importance weight). Existing methods occupy scattered, underexplored points on these two axes: IPPO treats both separately; MAPPO pairs a team-level advantage with per-agent ratios; HAPPO employs sequential ratios with per-agent advantages; and single-agent reductions operating on factorized joint policies aggregate both into fully joint products. We formalize these two design choices as support matrices $\SA$ and $\SR$, and prove a canonical structure: the expected multi-agent policy optimization objective depends on the pair $(\SA,\SR)$ only through their matrix product $\tS=\SR\SA$. This yields two key consequences: (i) Redundancy: the two support matrices are interchangeable with respect to the signal, meaning neither aggregation pattern is inherently superior.(ii) Variance Ordering: the advantage aggregates rewards as a sum (additive variance with an interior bias-variance optimum at the coupling neighborhood), whereas the ratio aggregates likelihood ratios as a product (multiplicative variance that grows exponentially with support size, with no accompanying bias reduction). The resulting design principle is unambiguous: aggregate neighbors in the advantage, sized to the coupling neighborhood, and keep the ratio per-agent.

Figures

Figures reproduced from arXiv: 2607.17924 by Sen Li, Zijian Zhao.

Figure 1
Figure 1. Figure 1: Bias, variance, and MSE of the multi-agent policy-gradient estimator on a single-step [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Advantage-support sweep across the four coupling families: final team return versus the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Agent-count scaling across the four coupling families. For each family and agent count [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Schematic of the four coupling families (illustrated at a reduced scale; nodes represent [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Advantage-support training curves across the four coupling families. Team return versus [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ratio-support redundancy under standard PPO across the four coupling families. Team [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Traffic-signal control on a 196-intersection network. Advantage-support training curves (per-agent ratio; mean over 3 seeds, ±1 std shaded bands) on all three metrics—team reward, average queue, and average waiting time—versus PPO iteration. Each panel compares three local advantage supports (independent (ρA = 0) and one/two-hop neighborhoods) against the global/team advantage (ρA = ∞). Insets zoom the las… view at source ↗
Figure 8
Figure 8. Figure 8: Ratio-support study on the 196-intersection network under a standard PPO configuration (mean over 3 seeds, ±1 std bands). Top row: the 1-hop neighborhood advantage held fixed. Bottom row: the independent (S A = I) advantage held fixed. Columns show team reward, average queue, and average waiting time versus PPO iteration. Within each panel, four curves vary the ratio support: per-agent (ρR = 0), 1-hop, 2-h… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 4 linked inside Pith

  1. [2]

    Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533,

    Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533,

    Pith/arXiv arXiv 2011

  2. [8]

    Zijian Zhao, Jing Gao, and Sen Li

    URL https://openreview.net/forum?id=symgW6FhA6. Zijian Zhao, Jing Gao, and Sen Li. Bridging marl to sarl: An order-independent multi-agent transformer via latent consensus.arXiv preprint arXiv:2604.13472,

    Pith/arXiv arXiv

  3. [11]

    and difference-reward methods (Wolpert & Tumer, 2001; Li et al.,

    2001

  4. [12]

    introduces a spatial discount factor that down-weights distant agents’ rewards inside each local return—an advantage support tapered by graph distance. We observe an asymmetry in this literature: a rich set of methods aggregate the advantage locally, but we are not aware of any cooperative method that aggregates the ratio locally (e.g., a local importance...

    2026

  5. [1994]

    Scalable multi-agent reinforcement learning for networked systems with average reward.Advances in Neural Information Processing Systems, 33:2074–2086, 2020a

    Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward.Advances in Neural Information Processing Systems, 33:2074–2086, 2020a. Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. InLearning for Dynamics ...

    Pith/arXiv arXiv 2074

  6. [2011]

    Manhattan 28×7 grid (n= 196 traffic signals) via sumo-rl (Alegre, 2019), with the standard MLP-actor MAPPO from the toy suite; the only tunable parameter is the advantage support, built by k-hop expansion on the road-adjacency graph, with a per-agent ratio throughout. Each intersection i is an agent with a local observation oi (the standard sumo-rl encodi...

    2019

  7. [2018]

    A comprehen- sive survey on multi-agent cooperative decision-making: Scenarios, approaches, challenges and perspectives.arXiv preprint arXiv:2503.13415,

    Weiqiang Jin, Hongyang Du, Biao Zhao, Xingwu Tian, Bohang Shi, and Guang Yang. A comprehen- sive survey on multi-agent cooperative decision-making: Scenarios, approaches, challenges and perspectives.arXiv preprint arXiv:2503.13415,

    Pith/arXiv arXiv

  8. [2019]

    Sumo–simulation of urban mobility: an overview

    9 Preprint Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. Sumo–simulation of urban mobility: an overview. InProceedings of SIMUL 2011, the third international conference on advances in system simulation. ThinkMind,

    2011

  9. [2022]

    Markov games as a framework for multi-agent reinforcement learning

    Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157–163. Elsevier,

    1994

  10. [2024]

    13 B.2 Proof of Lemma 2 (multiplicative variance)

    APPENDIXCONTENTS A MAMDP Formulation 12 B Proofs 12 B.1 Proof of Lemma 1 (unbiased weight) . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.2 Proof of Lemma 2 (multiplicative variance) . . . . . . . . . . . . . . . . . . . . . 13 B.3 Proof of Theorem 1 (support factorization) . . . . . . . . . . . . . . . . . . . . . . 13 B.4 Proof of Corollary 1 (p...

    1994

  11. [2025]

    Generalized per-agent advantage estimation for multi-agent policy optimization.arXiv preprint arXiv:2603.02654,

    Seongmin Kim, Giseung Park, Woojun Kim, Jiwon Jeon, Seungyul Han, and Youngchul Sung. Generalized per-agent advantage estimation for multi-agent policy optimization.arXiv preprint arXiv:2603.02654,

    arXiv

  12. [2026]

    Trust region policy optimisation in multi-agent reinforcement learning

    JG Kuba, R Chen, M Wen, Y Wen, F Sun, J Wang, and Y Yang. Trust region policy optimisation in multi-agent reinforcement learning. InICLR 2022-10th International Conference on Learning Representations, pp

    2022